If you have ever watched an LLM training run crawl because the GPUs were starving for data, the bottleneck was almost never the matrix multiplies. It was the path between the GPUs, the network, and the storage. GPU Direct RDMA is the family of NVIDIA technologies that removes the CPU and the host memory from that path, letting data flow directly between GPU memory and a peer GPU, a NIC, or an NVMe drive. In a modern AI factory, GPU Direct is no longer a nice-to-have. It is the reason an 8-GPU node trains at 700 GB/s of all-reduce instead of 60 GB/s.

This guide walks through what GPU Direct RDMA actually does under the hood, how the four GPU Direct flavors (P2P, Storage, RDMA, and Async) fit together, what hardware you need on the NIC side, and how to design the fabric so that GPU Direct keeps paying off as you scale from one node to one thousand. We close with the five questions buyers ask most often when they specify a Mellanox / NVIDIA ConnectX, BlueField, or Quantum switch for a new GPU cluster, and how those answers map to products we ship at Zhongxin Data.

Table of contents

What GPU Direct RDMA actually is

GPU Direct is a set of NVIDIA kernel and driver features that let a third-party device DMA directly into or out of pinned GPU memory. “Third-party device” here means any peer that can issue a PCIe or NVLink peer-to-peer transaction: a sibling GPU on the same root complex, a network interface card, or an NVMe controller with the right firmware.

Without GPU Direct, a typical GPU-to-GPU transfer across nodes looks like this:

  1. Source GPU writes tensor to host DRAM (through PCIe).
  2. Host CPU copies the buffer into a pinned socket buffer.
  3. NIC’s DMA engine reads that socket buffer and posts it onto the wire.
  4. Remote NIC writes into its host DRAM.
  5. Remote CPU copies the buffer into the destination GPU.
  6. Destination GPU reads the data from host DRAM (through PCIe again).

With GPU Direct RDMA, steps 1, 2, 4, and 5 collapse. The source NIC DMAs straight from GPU memory; the destination NIC DMAs straight into GPU memory. The CPU and the host memory buffer are still in the picture for control plane operations, but they are no longer on the data path.

The headline number: GPU Direct RDMA removes roughly 60–80% of the latency of an equivalent host-mediated transfer, and it nearly doubles the achievable streaming throughput because you stop bouncing the same buffer across the PCIe root complex twice.

The four flavors of GPU Direct

NVIDIA bundles four related features under the GPU Direct brand. They are independent but composable, and most production clusters use all four at once.

GPU Direct feature comparison
Feature Source Destination Typical use case Protocol
GPU Direct P2P GPU Peer GPU (same node) NCCL all-reduce inside one box NVLink, PCIe P2P, BAR
GPU Direct RDMA GPU Remote GPU (across node) Distributed training collectives InfiniBand RC, RoCE v2
GPU Direct Storage NVMe / parallel file system GPU Checkpoint / dataset streaming NVMe-oF, NFS, SCSI, DAOS
GPU Direct Async NIC / DPU GPU Kernel-bypass data placement into GPU DPDK, DOCA, GPUAsync

Two of these deserve a closer look because they are the ones that drive network design.

GPU Direct RDMA (cross-node collectives)

This is the one that consumes the entire InfiniBand or RoCE fabric. NCCL detects that the source and destination buffers both live in GPU memory, sees that the NIC supports GPU Direct, and issues an RDMA write that points directly at the destination GPU’s BAR-mapped memory. The wire transaction looks exactly like any other RC RDMA write; what is different is that the PCIe transaction on the remote end terminates inside the GPU, not inside system memory.

The two protocols supported are:

  • InfiniBand Reliable Connected (RC) — the default for HDR 200 Gb/s and NDR 400 Gb/s fabrics.
  • RoCE v2 (RDMA over Converged Ethernet) — the default for Spectrum-X and Ultra Ethernet fabrics, with PFC + ECN for lossless behavior.

GPU Direct Storage (dataset and checkpoint I/O)

GPU Direct Storage extends the same trick to NVMe and parallel file systems. Instead of staging a 100 GB checkpoint through the CPU and host memory, the DPU or NIC DMAs straight from the NVMe controller’s submission queues into the target GPU. NVIDIA has demonstrated end-to-end bypass pipelines from local NVMe into a single H100 at over 90% of the device’s peak bandwidth.

The fabric implications are subtle. GPU Direct Storage does not require a special network adapter on the GPU side, but it does require:

  • NVMe-oF target support on the storage array (or a parallel file system like Lustre / GPFS / BeeGFS / DAOS).
  • A NIC or DPU that can act as a PCIe peer and bypass the kernel (ConnectX-6 Dx and later, or BlueField-2 / BlueField-3).
  • Pinned GPU memory on the destination side, owned by the consumer CUDA stream.

Why bypassing the CPU matters at LLM scale

It is worth being precise about the cost of the host-mediated path, because at 70-billion-parameter scale the data path is the whole job.

Cost of one GPU-to-GPU all-reduce at H100 NVL, 8 GPUs per node
Path Latency (one hop) Sustained BW (8 GPUs) CPU cores burned
CPU-mediated (memcpy + send) ~22 µs ~60 GB/s 4–6
GPU Direct P2P (NVLink) ~3 µs ~900 GB/s 0
GPU Direct RDMA (NDR 400G) ~5 µs ~720 GB/s 0

What this table hides is the secondary cost: every CPU-mediated transfer leaves interrupts, context switches, and cache pollution behind. On an 8-GPU H100 node, doing all-reduce through the CPU costs roughly 30–40% of one socket’s throughput for the duration of the collective, and that overhead scales with model size. Removing the CPU from the data path is what gives modern NCCL ring and tree algorithms room to actually achieve their theoretical bandwidth.

Why this matters now: Mixture-of-experts models with hundreds of experts per layer are dominated by all-to-all collectives. Expert parallelism turns a previously compute-bound step into a network-bound step, and the difference between CPU-mediated and GPU Direct RDMA on an all-to-all is the difference between a usable training run and an unusable one.

Hardware requirements: NICs, switches, PCIe, GPUs

GPU Direct is not magic. There are five pieces of hardware that all have to line up. Get any one of them wrong and you silently fall back to the slow path.

  1. NVIDIA / Mellanox ConnectX or BlueField NIC. ConnectX-5 supports GPU Direct RDMA over IB and RoCE. ConnectX-6 Dx adds hardware-accelerated storage and is the floor for new designs in 2026. ConnectX-7 adds 400 Gb/s and is the right pick for NDR fabrics. BlueField-2 / BlueField-3 add an Arm core that can host GPU Direct Async offloads.
  2. Matching switch fabric. For InfiniBand: Quantum-2 (HDR, 200 Gb/s) or Quantum-3 (NDR, 400 Gb/s) with managed subnet manager. For Ethernet: Spectrum-3 or Spectrum-4 with adaptive routing and RoCE congestion control.
  3. PCIe at Gen4 x16 or better. A ConnectX-7 running at PCIe Gen3 x16 is bandwidth-starved by roughly 35% relative to a Gen4 x16 slot. Gen5 is preferred for new builds; Gen6 unlocks 800 Gb/s adapters cleanly.
  4. GPU with large BAR support. A100, H100, H200, B100, B200, GB200 all expose enough BAR1 space for peer-to-peer DMA. Consumer cards (RTX 4090 and below) do not and will silently fall back.
  5. NUMA-aware host platform. The NIC must sit behind the same root complex as the GPUs that will DMA through it. Pinning NCCL to the wrong NUMA node is the single most common cause of “GPU Direct mysteriously doesn’t work” tickets.

Browse our Network Adapters collection for ConnectX-5, 6 Dx, 7, and BlueField-2 / BlueField-3 cards in both used condition, all flashed to the latest firmware and validated for GPU Direct.

Software stack: NCCL, UCX, OFED, DPDK

The hardware is the easy part. The software is where most teams lose a week the first time.

GPU Direct software stack
Layer Component What it does Notes
Application PyTorch / JAX / Megatron Issues collective ops Uses NCCL backend by default
Collective library NCCL Topology-aware rings / trees Auto-detects GPU Direct when peer buffers are pinned
Transport UCX / libibverbs / libxrc RDMA primitives UCX is the default for HPC, NCCL uses verbs directly
NIC driver / firmware MLNX_OFED User-space verbs, GPU Direct shim Match OFED version to NIC firmware exactly
Subnet manager (IB) OpenSM / UFM Routing, congestion maps Required for adaptive routing on Quantum switches
Subnet manager (Eth) Cumulus / SONiC + DCC PFC + ECN, QoS Required for lossless RoCE

Three rules of thumb keep this stack stable in production:

  • Pin the OFED version to the firmware version. Mixing MLNX_OFED 23.10 with NIC firmware 28.40 is the leading cause of subtle GPU Direct RDMA timeouts.
  • Run a managed subnet manager. OpenSM in stand-alone mode is fine for one rack. Once you go past two racks, run NVIDIA Unified Fabric Manager (UFM) so that adaptive routing and SHARP can coordinate across the fabric.
  • Test with NCCL all-reduce benchmarks before declaring victory. The nccl-tests all_reduce_perf binary will tell you in 60 seconds whether you are on the GPU Direct fast path or the slow path.

Sizing the fabric for GPU Direct

The easiest sizing mistake is to compute the bandwidth you need for a single all-reduce and forget that expert parallelism, pipeline parallelism, and checkpoint I/O all hit the same fabric at the same time.

A practical rule of thumb for a 2026 AI training cluster:

  • Per-GPU NIC bandwidth: 1× 400 Gb/s (NDR) or 1× 200 Gb/s (HDR) for the bulk of training jobs.
  • Per-GPU NIC bandwidth, frontier models: 2× 400 Gb/s on the rail-optimized hosts (front-end GPUs carry more cross-rack traffic).
  • Storage tier: a separate BlueField-3-connected storage VLAN, sized for 2× the checkpoint size in GB/s, so that checkpointing does not stall the next iteration.

For a 1,024-GPU cluster, this typically maps to a 2:1 oversubscribed fat-tree of Quantum-2 or Quantum-3 switches, with adaptive routing enabled and congestion control tuned for the expected mix of collectives.

Cost and refresh cadence

The honest cost story is that a properly sized GPU Direct fabric costs about the same as the GPUs it connects. A 1,024-GPU H100 cluster with NDR fabric will spend roughly $0.18–$0.22 per GPU per hour on the network alone, amortized over a four-year life. The good news is that the fabric is the most reusable part of the cluster: a 2023 NDR fabric will happily carry B200 and GB200 GPUs at full speed for several generations.

This is where used InfiniBand switches and ConnectX adapters start to make economic sense for capacity-tier clusters. We ship used Quantum-2 switches with verified firmware, full burn-in, and the same warranty window as new units, typically at 40–60% of the OEM list price. For inference clusters and capacity-tier training, this is the single biggest TCO lever in the design.

Rule of thumb: if a cluster is doing capacity training, inference, or model evaluation, used HDR/NDR fabric is the right answer. If it is doing frontier pretraining, new NDR/XDR fabric pays for itself in faster time-to-first-checkpoint.

Frequently asked questions

Is GPU Direct RDMA the same as InfiniBand RDMA?

No, but they are closely related. RDMA is the general capability (remote DMA without involving the host CPU on the data path). GPU Direct RDMA is NVIDIA’s specific implementation of RDMA where the source or destination buffer lives in GPU memory. You can run GPU Direct RDMA over InfiniBand (RC transport) or over RoCE v2 on Ethernet; both are supported and both bypass the CPU in the same way.

Do I need InfiniBand to use GPU Direct RDMA?

No. RoCE v2 on a lossless Ethernet fabric (Spectrum-3 / Spectrum-4 with PFC + ECN) gives you GPU Direct RDMA at line rate. The trade-off is that you have to tune the Ethernet fabric carefully — even one drop per million packets will tank collective throughput. InfiniBand is easier because lossless behavior is built into the link layer; Ethernet requires you to build losslessness with priority flow control.

Will GPU Direct work with consumer GeForce cards?

Generally no. CUDA P2P across GPUs and GPU Direct RDMA both require peer BAR mapping into the GPU’s address space. Datacenter cards (A100, H100, H200, B100, B200, GB200, L40, L4) expose the necessary BAR1 aperture. Consumer GeForce cards (RTX 4090 and below) do not. The driver will silently fall back to the host-mediated path, which works but loses most of the benefit.

What is the difference between GPU Direct RDMA and GPU Direct Storage?

GPU Direct RDMA is about GPU-to-network: moving data between GPUs across nodes without CPU involvement. GPU Direct Storage is about storage-to-GPU: moving data from local NVMe or an NVMe-oF target into GPU memory without CPU involvement. They share the same low-level kernel and driver shims, but they are configured and tuned independently. Most modern training pipelines use both: GPU Direct Storage for the dataset / checkpoint pipeline, GPU Direct RDMA for the collectives.

How do I know if my cluster is on the GPU Direct fast path?

Run nccl-tests/all_reduce_perf -b 1G -e 8G -f 2 -g 8 on a single node, then across two nodes. If the per-GPU bus bandwidth on the cross-node run is within 10–15% of the intra-node NVLink run, you are on the fast path. If it is more than 30% lower, you are either falling back to host memory (check NCCL_DEBUG=INFO for “P2P” vs “NET” lines) or your NUMA pinning is wrong.

If you are designing or refreshing a GPU cluster, these guides cover the rest of the stack:

Browse our Network Adapters for ConnectX-5 / 6 Dx / 7 and BlueField cards, InfiniBand Switches for Quantum-2 / Quantum-3 fabric, and Used Network Switches for capacity-tier builds. Need help sizing the fabric? Contact our solutions team on WhatsApp with your GPU count, model, and topology target.