By Zhongxin Data · Technical Guides · September 13, 2026

The network is no longer a utility in AI factories — it is the deterministic substrate that decides whether 10,000 GPUs finish a training run in 18 days or 31 days. Spine-Leaf with predictable Clos behaviour is the most deployed topology to make that bet.

If you are provisioning an AI cluster of any meaningful scale in 2026 — a single 64-GPU pod, a 1,024-GPU row, or a multi-rack 16,384-GPU supercomputer — you are almost certainly running a two-tier Spine-Leaf (Clos) fabric, possibly extended to a three-tier super-spine when pods reach tens of thousands of endpoints. This guide is a working engineering reference: it walks through the bandwidth math, oversubscription ratios, ECMP and adaptive routing choices, buffer-depth planning, optics/SerDes co-design, and the failure modes that bite at scale.

We will reference the Mellanox / NVIDIA Quantum and Spectrum switch families we stock at zhongxindata.hk (Quantum-2 QM9700, Quantum-3 QM3400/QM3500, Spectrum-4 SN5600, Spectrum-3 SN3700/SN4700) — because the design choices below are best understood when tied back to the silicon that actually runs them.

1. Why Clos topology won the AI fabric debate

Three topologies dominate hyperscale and AI fabrics today:

  • Spine-Leaf (two-tier Clos) — every leaf connects to every spine; non-blocking between tiers.
  • Fat-Tree (k-ary n-tree, often three tiers) — generalization of Clos using identical commodity switches; classic supercomputing choice.
  • Dragonfly / Dragonfly+ — group-level all-to-all with long-haul links between groups; minimizes switch count, maximizes optical reach.

For AI training, Spine-Leaf wins on three counts: (a) predictable latency because the maximum hop count is fixed at two switch hops between any two endpoints inside the same pod; (b) graceful incremental scaling — add a leaf, add a spine, repeat; (c) commercial silicon (Spectrum-4, Quantum-2/3) is now purpose-built for it with deep shared packet buffers and adaptive routing features tuned for AI collectives.

2. Building the bandwidth model

The job of the fabric designer is one number: does the worst-case collective fit in the time budget? For an all-reduce across N GPUs using a ring or tree algorithm over RoCE or IB, the per-step time is bounded by the slowest link. Get the spine uplink oversubscription wrong and your 10,000-GPU cluster behaves like a 1,000-GPU cluster with a 10× tail latency problem.

2.1 Oversubscription cheat sheet

Workload Recommended Leaf-to-Spine Oversubscription Why
Hyperscale AI training (LLM, MoE) 1:1 (non-blocking) Collectives saturate east-west. Any oversub shrinks effective bandwidth.
HPC simulation (CFD, FEA) 1:1 to 1.2:1 Bursty collectives tolerate slight oversub if buffer is deep.
Mixed training + inference 1.5:1 East-west dominates, but inference east-west is less bursty.
General-purpose cloud / multi-tenant 3:1 to 5:1 Tenants do not all blast at once; cost wins.
Storage (NVMe-oF / Ceph) 1:1 Loss = retransmits = tail latency = unhappy storage.

For AI-only pods the rule is: budget non-blocking, then fight capex. We routinely see 1:1 fabrics where the leaf-to-spine uplinks are QSFP-DD 800G and the spines are QM9700 64-port 400G HDR — that’s 1:1 oversubscription at the port level.

2.2 A worked example — 1,024-GPU pod

Assumptions: 8 GPUs per host (H100/MI300), dual-port ConnectX-7 400G NICs in an LACP-ish active-active mode but effectively using one NIC per rail. Each leaf switch has 32 hosts × 2 NICs = 64 downlinks. 8 leaf switches × 32 hosts = 256 hosts = 2,048 GPUs. That’s only half a pod — double it to 16 leaves for 4,096 GPUs, and you need 16 spine switches if you want 1:1. With Spectrum-3 SN3700 (32× 400G) for leaves and Quantum-2 QM9700 (64× 400G HDR) for spines, you build the classic 16×16 = 256-port non-blocking Clos fabric.

Tier Switch Port count used Speed Aggregate
Spine QM9700 (Quantum-2) 64 × downlinks + 0 uplinks 400G (NDR) per port 25.6 Tbps
Leaf SN3700 (Spectrum-3) 32 × server + 16 × spine 400G down / 400G up 12.8 Tbps uplink = 1:1
Server NIC ConnectX-7 2 × PCIe Gen5 x16 400G 400 Gbps per rail

3. ECMP, adaptive routing, and the dark art of “spraying” collectives

Clos relies on ECMP (Equal-Cost Multi-Path) to spread flows across multiple equal-cost spines. Standard ECMP hashes on the 5-tuple, which works fine for many small flows but penalizes the single elephant flow of an all-reduce: only one spine gets hit.

This is where Adaptive Routing earns its name. NVIDIA Quantum-2 introduced out-of-order spray (packet-by-packet spraying across healthy paths) specifically so that a single NCCL all-reduce uses every spine link simultaneously. Spectrum-4 with SHARP-like behaviour, plus RoCE’s explicit congestion notification (ECN), gets most of the same benefit on Ethernet fabrics.

Design rule of thumb: If your AI job is dominated by > 80% NCCL collectives, adaptive routing + ECN gives you back 25–40% of fabric time. If your job is mostly parameter-server gradient exchanges, standard ECMP is fine.

4. Buffer depth: why it matters more than peak throughput

Marketing materials obsess about Tbps. Real-world Clos fabrics fail because of buffer. During a 16K-GPU all-reduce the incast pattern at any leaf can briefly oversubscribe the destination by 10–50×. Without deep shared buffers, packets drop, TCP/RoCE backs off, and you lose minutes of training time.

Switch Shared Buffer Per-port @ 400G Incast Tolerance
Spectrum-3 SN3700 32 MB shared ~5 ms at line rate Good for < 8K GPU
Spectrum-4 SN5600 160 MB shared ~25 ms at line rate Strong to 32K GPU
Quantum-2 QM9700 ~120 MB shared ~18 ms Tuned for HPC/AI incast
Quantum-3 QM3500 ~270 MB shared ~40 ms Best-in-class incast tolerance

At scale (8K+ GPUs), Spectrum-4 SN5600 or Quantum-3 are the only sensible choices. Choosing a switch purely on port count without checking buffer depth is one of the most expensive mistakes in AI fabric design.

5. Optics, SerDes, and the 800G → 1.6T transition

The fabric speed roadmap is:

  • HDR (200G per lane, 400G port): Mellanox Quantum-2, ConnectX-7. Mature in 2026.
  • NDR (400G per lane, 800G port): ConnectX-8 / BlueField-3, Quantum-3. Production at hyperscalers in 2025-2026.
  • XDR (800G per lane, 1.6T port): ConnectX-9, Quantum-4 — sampling 2026, volume 2027.

For the switch-to-server segment, the form factor in 2026 is split:

  • OSFP for 800G NDR (16×50G PAM4 = 800G; can be split to 2×400G via breakout).
  • QSFP-DD112 for 800G in data centers that standardized on QSFP-DD originally.
  • OSFP 1600 for 1.6T XDR — new connector, requires pull-tab / heat-sink redesign.

If you are buying greenfield in 2026, specify OSFP-capable switches. If you are extending an existing QSFP-DD estate, stick with QSFP-DD112 modules for one more generation. Zhongxin Data carries both form factors — for example the SN3700 Spectrum-3 32-port 400G switch and the ConnectX-7 400G adapter family are the proven 2026 AI fabric building blocks.

6. When to scale up vs scale out

Two-tier Clos stops scaling practically at ~256 ports per tier, which means ~65,000 endpoints if you push it. For 100,000+ GPU AI factories (xAI Colossus 100K, Meta multi-MW pods), you need a three-tier super-spine or move to Dragonfly+.

Scale Topology Switch Tier 1 / Tier 2 Typical silicon
256–4,096 GPUs Spine-Leaf (2-tier Clos) Leaf / Spine SN3700 + QM9700
4,096–32,768 GPUs Spine-Leaf + Pods Leaf / Spine / Super-Spine SN5600 + QM9700 / QM3500
32,768–131,072 GPUs 3-tier Clos or Dragonfly+ Leaf / Spine / Super-Spine or Group SN5600 + QM3500 (Ethernet) or Quantum-3 with long-reach optics
> 131,072 GPUs Dragonfly++ / Custom Group routing w/ multi-link planes Custom or QM-class optimized

7. Operational gotchas at scale

Things that don’t show up in lab test reports but kill production:

  • MTU alignment. Set jumbo MTU (typically 9216) end-to-end. One mismatched vSwitch or VPC costs you 30% throughput.
  • PFC and ECN tuning. Priority Flow Control on RoCE needs careful queue mapping; default ON is rarely right.
  • Subnet Manager pinning on InfiniBand. Don’t let it move with chassis failures; pin to a paired pair of managed SM hosts.
  • Cable patching discipline. Label, label, label. A wrong-polarity MPO in a 1,024-port fabric will find you at 3am.
  • SNMP / telemetry overload. At 65k ports, naive polling melts your NMS. Use streaming telemetry (gNMI / gNTC) or sFlow cut-through sampling.

8. FAQ — Spine-Leaf & Clos for AI

Is two-tier Clos always non-blocking?

Yes, by definition — a two-tier Clos where every leaf downlinks equals the sum of all leaf-to-spine uplinks is a non-blocking fabric. Practical fabrics at AI scale use 1:1 oversubscription for east-west traffic. Storage and mixed-tenancy fabrics may accept 2:1 or higher oversubscription to save capex.

Why use InfiniBand at all when Spectrum-4 Ethernet is so capable?

Quantum-3 / Quantum-4 with SHARP gives you in-network aggregation that no Ethernet ASIC currently matches — up to ~3× effective throughput on all-reduce. For workloads where every microsecond of collective time matters (LLM training, large-scale HF simulations), IB still wins. For mixed AI + cloud + storage, Spectrum-4 with ECN/RoCE gets close enough that capex and operational simplicity tip the decision to Ethernet.

Do I really need 1:1 oversubscription?

For pure AI training pods, yes. A 256-GPU cluster with 2:1 oversubscription will hit > 80% saturation on all-reduce and burn 20–40% of your training time in queue. For mixed training + inference or storage, 1.5–2:1 is a reasonable compromise.

How many switch SKUs do I need to stock?

Most operators standardize on two: one leaf (e.g., SN3700 32×400G) and one spine (e.g., QM9700 64×400G). Large operators add a third: SN5600 for super-spine tier. Keeping the SKU count low reduces spare-part inventory and operator training cost.

What’s the typical lifecycle of a Clos fabric?

5–7 years. First generation (HDR / 200G per lane) tends to live 3–4 years before the optics become the bottleneck, then a mid-life uplifts to NDR / XDR by replacing optics and NICs only — switches are usually forward-compatible to one generation of optics.

9. Where to buy the building blocks

Zhongxin Data stocks the full Mellanox / NVIDIA Spectrum and Quantum switching portfolio, including sealed and bench-tested units, alongside ConnectX-7/8 adapters and OSFP/QSFP-DD optics that match the design rules above.

Need help designing your AI fabric? WhatsApp +86 13068616914 for sizing, BOM, and bench-tested pricing on multi-rack orders.