How NVLink, RDMA rails and GPUDirect Storage fit together

NVIDIA's H200 specification table lists two interconnect figures for the same GPU: 900 GB/s for NVLink and 128 GB/s for PCIe Gen5. NVLink has about seven times the bandwidth, so a GPU server ends up with more than one data path, and each path is designed separately.

The design below is for eight-GPU HGX H200 servers running Kubernetes with the NVIDIA GPU Operator, 400 Gb ConnectX-7 network cards, RoCE v2 and an all-flash NFS store. I cover the node and host side here and leave out the leaf and spine topology.

The three data paths of one HGX H200 server. Inside the chassis, path 1 is GPU to GPU: NVSwitch chips on the baseboard connect all eight GPUs, which are SXM modules, over NVLink at 900 GB/s, with copper traces and no cables, so every GPU reaches every other GPU at full bandwidth. Each GPU also connects over PCIe Gen5, 128 GB/s, peer-to-peer with no copy through host memory, to one of eight rails, which are 400 Gb ConnectX-7 ports. NVLink ends at the chassis wall. The eight rails leave the chassis into a RoCE v2 Ethernet fabric with jumbo frames and one lossless class. From the fabric, path 2 is node to node: collective traffic to other GPU servers, card to GPU memory with no kernel copy. Path 3 is storage to GPU: GPUDirect Storage, where a read over RDMA from the all-flash NFS store lands in GPU memory without a bounce through host RAM. — One HGX H200 server and where each of its three data paths ends.

NVLink and NVSwitch inside the chassis

The first path is GPU to GPU, and on an HGX server it never touches PCIe. The eight GPUs are SXM modules on one baseboard. NVLink runs as copper traces through that board into NVSwitch chips, with no cables. Because the links terminate on a switch, every GPU can reach every other GPU at full bandwidth at the same time.

That switch is why one model can be spread across all eight GPUs in a single box. It also means the whole server is built around the baseboard.

NVSwitch nodes also need the Fabric Manager service running from boot. Without it every CUDA call fails with error 802, even though nvidia-smi still reports healthy GPUs.

NVLink does not leave the chassis. Traffic to another node or to storage goes out through PCIe to a network card.

Eight rails between nodes

The second path is node to node. Each H200 server has eight ConnectX-7 ports dedicated to RDMA, and each one is a rail. GPU and card exchange data as PCIe peer-to-peer traffic, with no copy through host memory.

RDMA (remote direct memory access) lets the card read and write application memory without the kernel copying it. With GPUDirect that memory is GPU memory. This design runs it as RoCE v2 (RDMA over converged Ethernet, version 2) on jumbo frames.

It needs four settings on the hosts:

NVIDIA's open kernel modules. The DMA-BUF path that NVIDIA recommends over the older nvidia-peermem module requires them. With that path the in-kernel mlx5 driver is enough, and the hosts need no OFED.
PCIe ACS (access control services) disabled on the bridges between GPUs and network cards. If it is left on, peer-to-peer traffic detours through the root complex and transfers fail with retry-exceeded errors.
One flat subnet across all eight rails. Per-rail subnets are the textbook design. The flat subnet is simpler to operate and needs two ARP sysctls on every host.
The IOMMU in passthrough mode, with iommu=pt on the kernel command line. GPUDirect Storage needs it.

The lossless class has to match at every hop

RoCE can run on a network that drops packets, but it performs badly when it does. The usual design is one lossless traffic class configured consistently at every hop. Here it is priority 3 with priority flow control (PFC), selected by DSCP 26.

It has to be configured in three places. The sender marks its packets, which for NCCL, NVIDIA's collective communication library, is an environment variable. The host card maps DSCP 26 to priority 3 and enables PFC on it, which the dcb tool from iproute2 can set. The switch trusts DSCP on every host-facing interface and applies PFC and ECN to that queue.

The lossless class is set in three places. First, the sender marks the packet with DSCP 26, which for NCCL is an environment variable. Second, the host card maps DSCP 26 to priority 3 and enables PFC on that priority, set with the dcb tool from iproute2. Third, the switch trusts DSCP on every host-facing interface and applies PFC and ECN to that queue. With all three right the traffic is in priority 3 at every hop and no packets are dropped under congestion. With any one missed it is lossy at one hop or more, which raises no error and works until there is congestion. The check is an incast test, in which several senders converge on one receiver, because two hosts alone create no congestion. During the test, the per-queue byte counter on the switch and the per-priority counter on the card show which priority the traffic used. — The class is configured on the sender, the host card and the switch. During an incast test the counters show which priority the traffic used on the switch and on the card.

If any one of the three is missing, the traffic still flows, but it is no longer lossless end to end. A lossy RoCE fabric behaves normally until there is congestion. A bandwidth test between two hosts does not create any. Several senders converging on one receiver does.

For that reason the fabric acceptance test should be an incast test. While it runs, read the per-queue byte counter on the switch and the per-priority counter on the card, to confirm the traffic is in priority 3 at both. One counter is not enough, because a missing setting on the card does not show on the switch, and a missing setting on the switch does not show on the card.

GPUDirect Storage is a stack of four layers

The third path is storage to GPU. With GPUDirect Storage (GDS), a read from an NFS export over RDMA lands in GPU memory without a bounce through host RAM. It needs four layers, listed here from the application down:

cuFile, the library the application calls in place of POSIX reads and writes.
The nvidia-fs kernel module. The GPU Operator adds it as an extra container in the driver pod when gds.enabled is set, and current versions need the open GPU driver.
An NFS client whose RDMA transport has the nvidia-fs hook. The in-kernel rpcrdma module does not export the symbol nvidia-fs looks for. NVIDIA's documented route is the NFS-over-RDMA package in MLNX_OFED. On hosts without OFED, this layer is the storage vendor's own NFS client.
A mount with proto=rdma, defined as a second volume beside the existing TCP one so consumers opt in.

Setting gds.enabled changes the template of the driver DaemonSet, so the driver pod is replaced on every node. On a running cluster, plan for four things:

Put the driver upgrade skip label on every node first, then remove it one node at a time, each during a GPU outage for that node.
The driver pod builds the GPU driver and nvidia-fs every time it starts, so the node has to reach NVIDIA's apt repository at every start.
The device plugin restarts on all nodes as soon as the ClusterPolicy changes.
The operator uncordons the node when the driver upgrade finishes. If the node still needs work, cordon it again straight away, or the scheduler will place workloads on it.

Pods do not need to be privileged to use GDS. There are sixteen /dev/nvidia-fs* device nodes, and the default CDI handler injects them into the container.

GDS transfers fail when the host boots with the IOMMU in translated mode. In that mode the GPU memory range, which is physically contiguous, is mapped for the network card in 4 KiB pieces. nvidia-fs then rejects every transfer with DMA Address chunks 4096 != GPU Physical address chunks 1, and the write fails with EIO. NVIDIA's GDS guide says to disable the IOMMU on x86. iommu=pt is a reversible alternative, and transfers work with it.

The cuFile control path still succeeds with the IOMMU in translated mode. A smoke test that opens the driver and registers a file handle passes while every real transfer fails, so a useful test has to move data.

Layer three changes the most on the node. The vendor client replaces the node's entire NFS stack and is built for one exact kernel. It is easiest to build off the node in a container, with the version pinned. The source tarball ships both a Makefile and a makefile, so on a case-insensitive bind mount the build silently produces nothing. The client is a second component, after the GPU driver, that has to move in step with every kernel update.

Installing or updating the client needs a reboot, because reloading it in place unmounts every NFS filesystem on the node. A large GPU server is slow to reboot, and that reboot sets the outage for each node. If the kernel is updated and the client package is not rebuilt for it, the node boots with the in-tree NFS stack. Storage keeps working, GDS is off, and nothing reports an error.

cuFile also has a compatibility mode that falls back to ordinary POSIX I/O. Left on in production, it lets a GDS fault fall back to I/O through host memory while the workload stays up. With it on, a run that completes does not show that GDS was used. To check, turn compatibility mode off for a test transfer and watch the counters in /proc/driver/nvidia-fs/stats move.

The read and write counters in that file are off by default. Switch on rw_stats_enabled under /sys/module/nvidia_fs/parameters before reading it, otherwise they do not move. The test should also verify the data it wrote, because with compatibility mode off cuFile can return a short write.

GDS does not always raise throughput

GDS takes the CPU and host DRAM out of the data path. That does not guarantee more throughput than the path through host memory. Over an NFS RDMA mount the ceiling can sit in the path between the NFS RDMA transport and nvidia-fs, and a read into host memory followed by a copy to the GPU can be faster than GDS on the same mount.

Measure both paths on the same RDMA mount before deciding which one a workload uses. Run the comparison from a GPU that shares a PCIe switch with the rail card and from one that reaches it across NUMA nodes. If the result has the same shape for both, PCIe placement is not the limit.

What these choices cost in host security

Three of these choices weaken the node:

Secure Boot is off, because the vendor NFS client and nvidia-fs are unsigned out-of-tree modules.
The container that builds nvidia-fs runs privileged.
With IOMMU passthrough the kernel no longer uses the IOMMU to confine DMA from host devices. That includes the path between the network cards and GPU memory.

Should storage share the rails?

In this design the eight rails carry collective traffic and are also the route to storage. Because storage shares the rails, storage RDMA has to be in the lossless class too. The kernel NFS client connects through rdma_cm and sets no type-of-service value of its own, so its connections take a per-device default. That default has to be configured on every rail, otherwise storage goes out at priority 0 beside lossless collective traffic.

That default lives in the rdma_cm configfs tree, which is empty after every boot, so it has to be applied again at each boot or made persistent. It also covers only what the host sends. The storage array marks its own return traffic, so check the receive counter on the card as well as the transmit one.

Sharing also affects multipath. An NFS client can spread its connections across several rails, which puts storage load on the cards the GPUs use to talk to each other. Running storage on a single path avoids that, at the cost of storage bandwidth.

A workload on the RDMA mount does not necessarily use GDS. An inference KV cache such as LMCache can keep its storage tier on the RDMA mount, but when the cache server writes that tier from a tier in pinned host memory, the traffic is NFS over RDMA without GDS. The source cuFile sees is host memory, and the nvidia-fs counters do not move. Moving blocks directly between GPU memory and storage needs LMCache's --gds-l1-path variant, which replaces the host-memory tier.

Model weights can stay on the TCP mount when weight loading is not a bottleneck.

The rail split and the cache tier are both open choices in this design. If you have separated storage and collective traffic on an eight-rail node, I would like to hear how you divided the rails.