How NVLink, RDMA rails and GPUDirect Storage fit together

NVIDIA's H200 specification table lists two interconnect figures for the same GPU: 900 GB/s for NVLink and 128 GB/s for PCIe Gen5. NVLink has about seven times the bandwidth, so a GPU server ends up with more than one data path, and each path is designed separately.

This is the design we have deployed on a pair of eight-GPU HGX H200 servers. They run Kubernetes with the NVIDIA GPU Operator, 400 Gb ConnectX-7 network cards, RoCE v2 and an all-flash NFS store. I cover the node and host side here and leave out the leaf and spine topology.

NVLink and NVSwitch inside the chassis

The first path is GPU to GPU, and on an HGX server it never touches PCIe. The eight GPUs are SXM modules on one baseboard. NVLink runs as copper traces through that board into NVSwitch chips, with no cables. Because the links terminate on a switch, every GPU can reach every other GPU at full bandwidth at the same time.

That switch is why one model can be spread across all eight GPUs in a single box. It also means the whole server is built around the baseboard.

NVSwitch nodes also need the Fabric Manager service running from boot. Without it every CUDA call fails with error 802, even though nvidia-smi still reports healthy GPUs.

NVLink does not leave the chassis. Traffic to another node or to storage goes out through PCIe to a network card.

Eight rails between nodes

The second path is node to node. Each H200 server has eight ConnectX-7 ports dedicated to RDMA, and each one is a rail. GPU and card exchange data as PCIe peer-to-peer traffic, with no copy through host memory.

RDMA (remote direct memory access) lets the card read and write application memory without the kernel copying it. With GPUDirect that memory is GPU memory. We run it as RoCE v2 (RDMA over converged Ethernet, version 2) on jumbo frames.

On the hosts we made four choices:

• NVIDIA's open kernel modules. The DMA-BUF path that NVIDIA recommends over the older nvidia-peermem module requires them. With that path we use the in-kernel mlx5 driver and install no OFED on the hosts.
• PCIe ACS (access control services) disabled on the bridges between GPUs and network cards. If it is left on, peer-to-peer traffic detours through the root complex and transfers fail with retry-exceeded errors.
• One flat subnet across all eight rails. Per-rail subnets are the textbook design. The flat subnet is simpler to operate and needs two ARP sysctls on every host.
• The IOMMU in passthrough mode, with iommu=pt on the kernel command line. GPUDirect Storage needs it.

The lossless class has to match at every hop

RoCE can run on a network that drops packets, but it performs badly when it does. The usual design, and ours, is one lossless traffic class configured consistently at every hop. Ours is priority 3 with priority flow control (PFC), selected by DSCP 26.

It has to be configured in three places. The sender marks its packets, which for NCCL, NVIDIA's collective communication library, is an environment variable. The host card maps DSCP 26 to priority 3 and enables PFC on it, which we do with the dcb tool from iproute2. The switch trusts DSCP on every host-facing interface and applies PFC and ECN to that queue.

If any one of the three is missing, the traffic still flows, but it is no longer lossless end to end. A lossy RoCE fabric behaves normally until there is congestion. A bandwidth test between two hosts does not create any. Several senders converging on one receiver does.

For that reason our fabric acceptance test is an incast test. While it runs we read the per-queue byte counter on the switch and the per-priority counter on the card, to confirm the traffic is in priority 3 at both. One counter is not enough, because a missing setting on the card does not show on the switch, and a missing setting on the switch does not show on the card.

GPUDirect Storage is a stack of four layers

The third path is storage to GPU. With GPUDirect Storage (GDS), a read from an NFS export over RDMA lands in GPU memory without a bounce through host RAM. It needs four layers, listed here from the application down:

1. cuFile, the library the application calls in place of POSIX reads and writes.
2. The nvidia-fs kernel module. The GPU Operator adds it as an extra container in the driver pod when gds.enabled is set, and current versions need the open GPU driver.
3. An NFS client whose RDMA transport has the nvidia-fs hook. The in-kernel rpcrdma module does not export the symbol nvidia-fs looks for. NVIDIA's documented route is the NFS-over-RDMA package in MLNX_OFED. We run without OFED, so this layer is the storage vendor's own NFS client.
4. A mount with proto=rdma, defined as a second volume beside the existing TCP one so consumers opt in.

Setting gds.enabled changes the template of the driver DaemonSet, so the driver pod is replaced on every node. On a running cluster we had to plan for four things:

• We put the driver upgrade skip label on every node first, then removed it one node at a time, each during a GPU outage for that node.
• The driver pod builds the GPU driver and nvidia-fs every time it starts, so the node has to reach NVIDIA's apt repository at every start.
• The device plugin restarts on all nodes as soon as the ClusterPolicy changes.
• The operator uncordons the node when the driver upgrade finishes. We still had work to do on that node, and workloads landed on it within seven minutes, before we cordoned it again.

Pods do not need to be privileged to use GDS. There are sixteen /dev/nvidia-fs* device nodes, and the default CDI handler injects them into the container.

The IOMMU blocked GDS on every node until we changed it. Our Ubuntu 24.04 hosts booted with the IOMMU in translated mode. In that mode the GPU memory range, which is physically contiguous, was mapped for the network card in 4 KiB pieces. nvidia-fs rejected every transfer with DMA Address chunks 4096 != GPU Physical address chunks 1, and the write failed with EIO. NVIDIA's GDS guide says to disable the IOMMU on x86. We chose iommu=pt because it is reversible, and transfers work with it.

The cuFile control path still succeeds with the IOMMU in translated mode. A smoke test that opens the driver and registers a file handle passes while every real transfer fails, so our test has to move data.

Layer three changes the most on the node. The vendor client replaces the node's entire NFS stack and is built for one exact kernel. We build it off the node in a container and pin the version. The source tarball ships both a Makefile and a makefile, so on a case-insensitive bind mount the build silently produces nothing. The client is a second component, after the GPU driver, that has to move in step with every kernel update.

Installing or updating the client needs a reboot, because reloading it in place unmounts every NFS filesystem on the node. An H200 server takes about 15 to 20 minutes to reboot, which sets the outage for each node. If the kernel is updated and the client package is not rebuilt for it, the node boots with the in-tree NFS stack. Storage keeps working, GDS is off, and nothing reports an error.

cuFile also has a compatibility mode that falls back to ordinary POSIX I/O. We leave it on in production so a GDS fault falls back to I/O through host memory and the workload stays up. With it on, a run that completes does not tell us GDS was used. To check, we turn compatibility mode off for a test transfer and watch the counters in /proc/driver/nvidia-fs/stats move.

The read and write counters in that file are off by default. We switch on rw_stats_enabled under /sys/module/nvidia_fs/parameters before reading it, otherwise they do not move. The test also verifies the data it wrote, because with compatibility mode off cuFile can return a short write.

GDS did not raise throughput

We compared GDS with the path through host memory on an H200 server, using the same RDMA mount and all eight GPUs, and the path through host memory was faster. Through GDS, between GPU memory and storage, we reached about 2.5 GB/s for writes and 4.2 GB/s for reads per mount. Into host memory, with a copy between host and GPU, the same mount reached about 3.5 GB/s for writes and 7 GB/s for reads.

The result had the same shape for every GPU, whether it shared a PCIe switch with the rail card or reached it across NUMA nodes. That puts the ceiling in the path between the NFS RDMA transport and nvidia-fs, and rules out PCIe placement.

On these servers the benefit of GDS is that data bypasses the CPU and host DRAM, with no gain in throughput.

What we gave up on host security

Three of these choices weaken the node, and we accepted them knowingly:

• Secure Boot is off, because the vendor NFS client and nvidia-fs are unsigned out-of-tree modules.
• The container that builds nvidia-fs runs privileged.
• With IOMMU passthrough the kernel no longer uses the IOMMU to confine DMA from host devices. That includes the path between the network cards and GPU memory.

Should storage share the rails?

On the H200 servers the eight rails carry collective traffic and are also the route to storage. Because storage shares the rails, storage RDMA has to be in the lossless class too. The kernel NFS client connects through rdma_cm and sets no type-of-service value of its own, so its connections take a per-device default. We have to configure that default on every rail, otherwise storage goes out at priority 0 beside lossless collective traffic.

That default lives in the rdma_cm configfs tree, which is empty after every boot, so it has to be applied again at each boot or made persistent. It also covers only what the host sends. The storage array marks its own return traffic, so we check the receive counter on the card as well as the transmit one.

Sharing also affects multipath. An NFS client can spread its connections across several rails, which puts storage load on the cards the GPUs use to talk to each other. For now we run storage on a single path.

We also had to decide what goes on this path. The inference engine's KV cache, LMCache, keeps its storage tier on the RDMA mount. That is NFS over RDMA, and it does not use GDS. The cache server writes the storage tier from a tier in pinned host memory, so the source cuFile sees is host memory and the nvidia-fs counters do not move. Moving blocks directly between GPU memory and storage needs the --gds-l1-path variant, which replaces the host-memory tier, and we have not decided on it.

Model weights stay on TCP. They load at about 39 GiB per GPU in roughly 33 seconds, and weight loading is not a bottleneck for us.

I have not settled the rail split or the cache tier. If you have separated storage and collective traffic on an eight-rail node, I would like to hear how you divided the rails.