For nearly two decades the data center fabric debate has been framed as InfiniBand vs Ethernet. In 2026 the question is sharper than ever. Large language models with hundreds of billions of parameters, distributed training jobs that must keep thousands of GPUs in lock-step, and inference fleets pushing tokens at millisecond budgets — each one of these workloads pressures the network in ways that a generic 10 GbE closet switch never had to face. The wrong choice of fabric can quietly double your training time, push power and cooling budgets past their limits, and lock you into a vendor ecosystem that you cannot easily escape.
At Zhongxin Data we supply both sides of that ecosystem: NVIDIA Mellanox InfiniBand adapters and Quantum/Spectrum switches, and a deep catalog of Broadcom-, Intel-, and Marvell-based Ethernet gear. This guide is the comparison we walk our customers through before they commit a single dollar to a new cluster. It is vendor-neutral on the protocol question, and very specific on the numbers.
Quick Answer: When to Pick Which
- Pick InfiniBand if you run tightly coupled HPC or AI training, need the lowest possible tail latency at scale, and want NVIDIA’s vertically tuned stack (GPUDirect RDMA, SHARP, NCCL).
- Pick Ethernet if you run a mixed cloud, storage, and AI workload, want multi-vendor optics and switches, or are deploying 200G/400G/800G fabrics on a converged leaf-spine.
- Pick Spectrum-X or Ultra Ethernet if you want Ethernet’s openness but also need the lossless, low-latency behavior that AI fabrics demand.
The Underlying Protocols Are Not the Same Animal
InfiniBand is defined by the InfiniBand Trade Association (IBTA). It is a single, end-to-end specification that covers the physical layer, the link layer, the transport layer, the software verbs interface, and the management agents. Every compliant device — HCA, switch, cable, transceiver — has to pass IBTA’s interoperability logo tests. There is no “InfiniBand over someone else’s L2”; it is the L2.
Ethernet, in contrast, is just IEEE 802.3. It defines a physical and MAC layer. Everything that AI fabrics care about — lossless behavior, congestion signaling, RDMA semantics, multi-pathing — is bolted on top by other standards bodies and industry consortia:
- RDMA: RoCE v2 (RDMA over Converged Ethernet, RFC 8852) or iWARP.
- Congestion control: DCQCN, PFC, ECN, and now the new Ultra Ethernet (UE) fabric spec.
- High availability: EVPN-VXLAN, MLAG, MC-LAG.
- Storage: NVMe-oF, iSCSI, SRP.
That split is both Ethernet’s strength (you can mix and match) and its weakness (the parts do not always interop as cleanly as IBTA’s tested stacks).
Side-by-Side: The Numbers That Matter
| Attribute | InfiniBand (NDR / XDR) | Ethernet (RoCE v2 / Spectrum-X) |
|---|---|---|
| Standard link speeds | NDR 400 Gb/s, XDR 800 Gb/s (8x of 100G/200G PAM4) | 100G, 200G, 400G, 800G (IEEE 802.3ck, 802.3df) |
| Switch port latency (typical) | 90–150 ns (cut-through, IBTA spec) | 300–500 ns (store-and-forward or cut-through) |
| End-to-end GPU-to-GPU (typical 8-hop) | ~1.5–2.5 µs | ~3–6 µs (RoCE v2), ~2.5–4 µs (Spectrum-X with adaptive routing) |
| Native RDMA | Yes, mandatory in the spec | Via RoCE v2, requires lossless L2 (PFC + ECN) |
| Congestion control | Hardware, deterministic, per-packet | Software-assisted (DCQCN) or vendor-tuned (Spectrum-X) |
| Adaptive routing | Built into Quantum-2/Quantum-X switches | Optional in UE, optional per-vendor |
| SHARP / in-network compute | Yes (SHARPv3 in NDR) | No equivalent in standard Ethernet |
| Max fabric size (single subnet) | ~48k endpoints (NDR), scaling with fat-tree | Effectively unlimited via EVPN |
| Topology flexibility | Fat-tree mostly; Dragonfly+ supported | Any L2/L3 topology, including Clos and Dragonfly |
| Optical transceivers | IBTA-tested, often proprietary form factors | QSFP-DD / OSFP — broad multi-vendor supply |
| Software stack | OFED, UCX, NCCL, MPI — all from one vendor ecosystem | Same upper stack, but a wider choice of drivers and tuning knobs |
Latency rows above are based on vendor datasheets (NVIDIA Quantum-2, Spectrum-4) and independent benchmarks from MLPerf and the TOP500 submission notes. Real numbers in production depend on switch buffer depth, MTU, and whether adaptive routing is enabled.
What the AI Training Workload Actually Demands
Distributed training is the workload that has pushed this comparison back to the top of the agenda. A single training step in a transformer model passes activations and gradients across all participating GPUs. For a 1024-GPU job with tensor parallelism and pipeline parallelism, the network must deliver:
- Collective bandwidth on the order of TB/s across the job.
- Tail latency for all-reduce operations on the order of single-digit microseconds.
- Near-linear scaling efficiency: every doubling of nodes should give close to 2x throughput.
Three network features, more than any other, decide whether you hit those numbers:
1. RDMA, GPUDirect, and the Kernel-Bypass Path
RDMA lets one GPU’s memory be read or written by another GPU’s process without the data touching the host CPU or the kernel’s TCP/IP stack. NVIDIA GPUDirect RDMA goes one step further: the data goes directly between the GPU HBM and the network adapter’s DMA engine, bypassing the host entirely. This is the single biggest reason why Mellanox ConnectX-7 / ConnectX-8 adapters show up in nearly every modern AI cluster, whether the fabric underneath is InfiniBand or RoCE v2 Ethernet.
2. In-Network Compute (SHARP)
InfiniBand’s Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) moves all-reduce and broadcast operations off the endpoints and into the switch ASIC. In a 1024-GPU cluster, SHARPv3 can reduce collective traffic by 60–80% and cut the step time accordingly. There is no exact Ethernet equivalent. Spectrum-X and Ultra Ethernet add parts of the picture (congestion-aware routing, packet spraying) but not a full SHARP-style compute offload.
3. Adaptive Routing and Congestion-Free Fabrics
When one path in a fat tree gets hot, the fabric should re-route around it — at packet granularity, not flow granularity. InfiniBand switches have done this for years. Ethernet vendors have caught up: Spectrum-4 supports per-packet adaptive routing, and the Ultra Ethernet specification now standardizes a similar feature across vendors.
Ethernet Has Caught Up — Mostly
If you walked into a 2022 cluster, the gap between InfiniBand and Ethernet on AI workloads was wide enough that the choice was simple. By 2026, the picture is more nuanced:
- 400 Gb/s and 800 Gb/s Ethernet are now shipping in volume. NVIDIA’s Spectrum-4 and Broadcom’s Tomahawk 5 both deliver 51.2 Tb/s per chassis, with 64x 800G ports. You no longer pay a speed tax for picking Ethernet.
- RoCE v2 with proper PFC, ECN, and DCQCN tuning is a mature stack. We routinely see customers hit >95% of NCCL all-reduce bandwidth on well-built Ethernet pods.
- NVIDIA Spectrum-X is a specifically engineered Ethernet platform for AI. It is not standard Ethernet — it adds adaptive routing, congestion control, and direct NCCL hooks — but it is Ethernet under the hood, with full IP/EVPN support alongside the AI-tuned path.
- Ultra Ethernet (UE), ratified in 2024 and shipping in silicon from 2025 onward, is the industry attempt to standardize the Spectrum-X-style fabric across all Ethernet vendors. By mid-2026 there are working UE switch ASICs and adapter cards from at least three silicon vendors.
Rule of thumb in 2026: if your AI training cluster is larger than 256 GPUs and tightly coupled, you should still benchmark InfiniBand first. If you are in the 64–256 GPU range, or you have a mixed cloud+AI fabric, Ethernet (especially Spectrum-X or UE) is the easier choice — and the cost-per-port story is noticeably better.
Total Cost of Ownership: The Quiet Winner
Capex per port is the number most buyers start with, and on that metric Ethernet usually wins:
| Cost Component | InfiniBand (NDR) | Ethernet (400G/800G) |
|---|---|---|
| Adapter (HCA / NIC), per port | Higher (ConnectX-7/8 NDR) | Lower (ConnectX-8 EN, BlueField-3, Broadcom NetXtreme) |
| Switch port, per 400G equivalent | Higher (Quantum-2) | Lower (Spectrum-4, Tomahawk 5) |
| Optics (400G/800G) | IBTA-locked, narrower supply | QSFP-DD / OSFP, multi-vendor |
| Cabling (copper / AECs) | Limited, vendor-specific lengths | Broad third-party supply |
| Power per 400G port (typical) | ~12–15 W | ~10–14 W |
| Software / management | UFM / NetQ, included in stack | Vendor NMS, plus SONiC / Spine-Leaf options |
| Training, hiring, skills | Niche, harder to recruit | Plentiful, mainstream skill set |
| Multi-vendor optics & switches | No | Yes |
Where InfiniBand earns its premium is in operations and time-to-result. A cluster that comes up faster, runs cooler, and finishes a training run a few hours sooner pays back the fabric premium in months, not years. The TCO question is therefore workload-specific: a hyperscaler running always-on recommendation models amortizes fabric costs across many jobs and will choose Ethernet; a research lab running a single 8x H100 fine-tune of a 70B model will choose InfiniBand for the run-time savings.
What About Storage and HPC Workloads?
Not every cluster is an AI cluster. The other major network consumer is distributed storage:
- NVMe-oF over RoCE v2 is the dominant protocol for high-performance block storage. It runs over Ethernet and is mature. NVMe-oF over InfiniBand exists but is rarely chosen — the customers who need it usually already have InfiniBand for compute and want a single fabric.
- HPC MPI workloads (CFD, weather, genomics) have used InfiniBand for 20 years and still do, primarily because of the low-latency MPI collectives. Spectrum-X and UE are starting to displace InfiniBand in some HPC sites, but the migration is slow and depends on the MPI library (OpenMPI 5.x has better UE support than older releases).
Migration Paths and Hybrid Fabrics
You do not have to pick one fabric for the whole data center. Many sites we work with run a hybrid:
- InfiniBand pod for the largest training jobs (256+ GPUs, NCCL-heavy).
- Ethernet leaf-spine for everything else: inference, storage, development, management.
- Border leaf connecting both fabrics at the data center gateway, with route filtering to keep failure domains separate.
ConnectX-8 adapters and BlueField-3 DPUs can present both an InfiniBand port and an Ethernet port from the same card, which makes the hybrid more practical than it used to be.
Frequently Asked Questions
Is InfiniBand faster than Ethernet for AI training?
Yes, in most AI training benchmarks measured at the end of 2025, InfiniBand NDR/XDR still beats equivalently priced Ethernet in step time and scaling efficiency, especially above 512 GPUs. The gap is closing fast — Spectrum-X and Ultra Ethernet have closed most of it for pods up to 256 nodes — but on the largest jobs InfiniBand still wins by 10–25%.
Can I run GPUDirect RDMA over Ethernet?
Yes. GPUDirect RDMA is supported over both InfiniBand and RoCE v2 Ethernet. The same ConnectX-7 or ConnectX-8 adapter will work in either mode. The only requirement on the Ethernet side is that the L2 fabric is properly configured for lossless behavior (PFC + ECN) and that adaptive routing is enabled.
What is Ultra Ethernet (UE) and why does it matter?
Ultra Ethernet is an industry-standard fabric specification, ratified in 2024, that adds AI-friendly behavior — packet spraying, adaptive routing, low-latency congestion control — on top of standard 802.3 Ethernet. The first UE switch ASICs shipped in 2025. UE matters because it gives buyers a non-NVIDIA Ethernet AI fabric with guaranteed interop, rather than depending on a single vendor’s tuning.
Should I buy new InfiniBand or wait for XDR / 800 Gb/s?
If your cluster is being built today and you have a 3-year horizon, NDR (400 Gb/s) InfiniBand is the safe choice — it is mature, broadly available, and supported by every major AI software stack. XDR (800 Gb/s) is shipping but the adapter supply is still constrained and pricing is high. If you can wait until late 2026, the XDR ecosystem will be more competitive.
Does 800G Ethernet make InfiniBand obsolete?
No. Raw link speed has not been the deciding factor for several years. The deciding factors are tail latency, in-network compute (SHARP), and how tightly the fabric is integrated with NCCL. 800G Ethernet closes the speed gap but does not close the latency and integration gap. For the largest jobs InfiniBand will continue to win on step time. For the long tail of mid-size jobs, 800G Ethernet (especially Spectrum-X and UE) is the more practical choice.
How Zhongxin Data Can Help
We supply both sides of this question and have done so for hundreds of cluster builds. Our typical engagement looks like this:
- Workload survey — what models, what batch sizes, what parallelism strategy, what storage traffic.
- Topology design — fat tree, Dragonfly+, or leaf-spine, with port counts and oversubscription ratios.
- Bill of materials — adapters, switches, optics, cabling, with a working BOM for both fabric choices.
- Rollout — pre-imaged racks, on-site racking and stacking, NCCL validation tests.
Browse our current stock of InfiniBand adapters and switches, Enterprise Ethernet networking gear, and RAID controllers / HBAs. If you are starting a new cluster and want a side-by-side BOM for InfiniBand vs Spectrum-X Ethernet, contact our solutions team — typical reply time is one business day.
Zhongxin Data is an independent supplier of enterprise and data center network hardware, headquartered in Hong Kong and shipping worldwide. NVIDIA, Mellanox, Broadcom, Intel, and the respective logos are trademarks of their owners. This article is for informational purposes and reflects market conditions as of mid-2026.