Introduction: Why Liquid Cooling Is Now a Networking Problem

For most of the last forty years, “data center cooling” meant computer room air conditioning (CRAC) pushing cold air under a raised floor. That world is gone. The arrival of 400 W–1000 W CPUs, 700 W–1200 W GPUs, and 800 Gb/s switches has made per-rack power density climb from 10 kW to 60 kW, 100 kW, and even 200 kW+ in flagship AI clusters. Air can no longer carry that heat away economically, and the bottleneck is no longer the server — it is the network fabric wrapped around the server.

This article is a field-tested guide to deploying liquid cooling together with a high-density network fabric. We cover the technology options (cold plate, single-phase immersion, two-phase immersion), the impact on top-of-rack switches, InfiniBand adapters, DPU-equipped BlueField SmartNICs, and 800G optical transceivers, and the operational practices that separate a pilot from a 50 MW AI factory.

The new rule of thumb: if your rack power exceeds 30 kW, treat liquid cooling as a network project — because the heat removal path determines which NICs, switches, transceivers, and optical modules you can deploy.

1. Cooling Technologies Compared

There are three mainstream liquid-cooling approaches in 2026. Each has different implications for the network hardware installed in the same rack.

Technology Coolant Touches Rack Density Network Hardware Compatibility Typical Use Case
Air cooling (rear-door or in-row) None — fans + coils 10–30 kW All standard switches/QSFP-DD modules Legacy enterprise, low-density edge
Direct Liquid Cooling (DLC) / cold plate CPU, GPU, sometimes DIMMs and VRMs 40–80 kW ToR switches & NICs still air-cooled (front of rack) AI training, mainstream hyperscale
Single-phase immersion Entire server submerged in dielectric fluid 60–150 kW Switches require special immersion-rated models or stay in adjacent air-cooled rack Dense GPU pods, crypto/HPC farms
Two-phase immersion Entire server; fluid boils at component surface 100–250 kW Most switches NOT rated; pods are GPU-only Frontier-class HPC, prototype AI factories

The table above is not theoretical — it is the practical compatibility matrix that drives procurement. A cold-plate DLC deployment can still use a standard NVIDIA Quantum-2 QM9700 or Spectrum-4 SN5600 ToR switch sitting at the top of the rack, because the switch’s heat-generating ASIC and PSUs are air-cooled. In a two-phase immersion tank, the same switch cannot be submerged; it must live in a “cold aisle” rack one or two positions away, with longer DAC/AEC cables bridging to the immersed GPU sleds.

2. The Network Stack Under Liquid Cooling

A typical 2026 AI rack looks like this when laid out for DLC cooling:

Layer Hardware Cooling Path Typical Power Critical Procurement Note
Compute sled 8x H200/MI300X GPU + dual EPYC Turin / Granite Rapids Cold plate on CPU, GPU, DIMMs ~10–14 kW Verify cold-plate covers all DIMM slots, not just CPU
Network sled ConnectX-8 / BlueField-3 DPU per GPU sled Cold plate or air (model-specific) 25–35 W per port BlueField-3 runs hotter than ConnectX — check vendor thermal spec
ToR switch QM9700 (NDR) / SN5600 (800G) Forced air (front-to-back) ~1.5–2.2 kW Airflow direction MUST match rack hot/cold aisle; check “port-side intake” vs “port-side exhaust”
Spine switch CS9500 / SN5600 in spine row Forced air ~2.5–3 kW Separate hot/cold aisle row; 12–15 kW per spine rack
Optical transceivers OSFP 800G DR8 / 2xFR4 Air (at switch faceplate) ~16–20 W each High-density OSFP cages need 30+ CFM of directed airflow

The crucial insight: in a DLC rack, the compute is liquid-cooled but the network is still mostly air-cooled. Your cooling design must handle both heat streams — a 14 kW compute sled dumps heat into the facility water loop, while the adjacent ToR switch exhausts ~2 kW into the rack’s air stream. Mixing them carelessly creates thermal dead zones and forces the ToR fans into a high-RPM state that accelerates failure.

3. Coolant Distribution Units (CDUs): Sizing for AI Loads

A CDU is the heart of a DLC loop. It isolates the facility water (which may be 30 °C warm water from a cooling tower) from the IT coolant (which touches chips at 45–65 °C). Sizing the CDU wrong is one of the most common causes of AI cluster underperformance.

CDU Class Loop Capacity Typical Deployment Network Considerations
Sidecar / rack-level 40–80 kW Single AI rack None — self-contained
Row-level 200–600 kW 4–12 racks in a hot/cold aisle Plumbing above rack, restrict top-of-rack optical cabling
Facility-level 2–20 MW Full AI factory pod (50–200 racks) Manifold routing dictates aisle width; plan 1.2 m+ between hot/cold aisles

Two practical points from the field:

  1. Leave 20–30% thermal headroom in the CDU. AI workloads spike. A CDU sized to “average” rack power will throttle when eight adjacent racks run a collective-comms all-reduce at the same time. This is also when your SHARP / NCCL traffic peaks, which is precisely when you cannot afford thermal throttling on the ToR switch.
  2. Coolant temperature affects switch inlet air. Modern DLC designs run IT coolant at 40–50 °C. The “cold air” entering the ToR switch is therefore 35–45 °C, not the 20 °C of legacy data halls. Verify that your switch model is rated for an ASHARE A4 (45 °C) inlet — many older ToR models are only A3 (40 °C) and will derate throughput.

4. High-Density Rack Topology Considerations

Once you cross 50 kW per rack, the physical layout of the network becomes a first-class design constraint. Three principles have emerged from real deployments:

  • Keep ToR within 2 m of the GPU sleds. Beyond that, DAC/AEC cable reach becomes the limiting factor at NDR 400G speeds, and AOC fiber costs balloon. This is why immersion-cooled pods often use an adjacent “switch rack” pattern rather than co-locating the switch in the tank.
  • Plan for 1U or 0U ToR, not 4U chassis. Modern AI fabrics favor fixed-configuration 1U 64-port ToR switches (QM9700, SN5600) over modular chassis. They consume less rack space, have shorter airflow paths, and can be replaced unit-by-unit during maintenance without disrupting an entire rack.
  • Optical cabling must be serviceable from the cold aisle. With 200+ fibers per rack, MPO/MTP trunk density becomes unmanageable from the rear. Spec “port-side intake” switches so all patching happens at the cold aisle face.

5. Operational Realities: What Trips Up First

Across multiple AI factory builds, the failure modes that hit first are predictable:

Issue Root Cause Network Impact Mitigation
ToR switch fans running at 100% RPM Inlet air too warm (A3 vs A4 inlet spec) PFC pause storms under thermal throttle Specify A4-rated switches; raise cold-aisle setpoint
Intermittent 800G optical link flaps OSFP cage thermal runaway Collective-comms timeouts, job restarts Increase directed airflow to faceplate; re-seat modules
Coolant leak trips IT-side loop Quick-disconnect failure on cold plate Entire rack goes dark; AI job loss Leak sensors + automatic isolation valves per rack
BlueField-3 DPU overheating DPU not on the cold-plate manifold Storage offload and OVS throughput drop 30–50% Confirm vendor DLC kit covers DPU; add heatsink if not

6. Used Hardware in Liquid-Cooled Racks

Many buyers assume that a DLC AI deployment must use brand-new switches and adapters. In practice, used ConnectX-6/7 adapters and Quantum-2 switches are deployed every day in cold-plate DLC racks, with no degradation in thermal performance. The key checks:

  • Used switches must be fitted with new fans and retested — old fan bearings fail faster at elevated cold-aisle temperatures.
  • Used transceivers must have fresh DOM (digital optical monitoring) calibration; aging laser bias currents drift outside spec after years of storage.
  • Adapter cards from previous-generation hosts (e.g., ConnectX-6 Dx on a PCIe Gen4 system going into a Gen5 DLC sled) work fine, but verify the host’s PCIe lane count — Gen5 x16 is required to feed 400G NDR at line rate.

If your procurement team is budget-constrained, a hybrid new/used approach — new ToR switches, used adapters and DAC cables — can deliver the same fabric performance at 40–60% of the all-new capex.

7. Frequently Asked Questions

Can I air-cool an 800G ToR switch in a 100 kW rack?

Yes — but only if the rack’s cold-aisle airflow is preserved. Modern 1U 64-port 800G switches like the SN5600 pull 1.8–2.2 kW and require roughly 200 CFM of directed airflow at 45 °C inlet. The common mistake is enclosing the switch in a “hot box” with the GPU sleds, which starves it of cold air. Keep the ToR at the top of the cold aisle, even if that means a 2–3 m DAC cable to the topmost GPU sled.

Do ConnectX-8 / BlueField-3 DPUs need cold plates?

ConnectX-8 NDR adapters (≤21 W per port) are typically air-cooled with a tall heatsink. BlueField-3 DPUs (≤75 W under heavy offload) often need a vendor-supplied cold-plate bracket, especially in 1U sleds where airflow is constrained. Confirm with the sled OEM before assuming an air-cooled install.

Is immersion cooling compatible with optical transceivers?

Standard QSFP-DD and OSFP modules are not rated for submersion. In immersion tanks, the network must live in an adjacent air-cooled rack, connected via fiber or AEC to the immersed GPU sleds. This is why most single-phase immersion deployments still top out at 60–100 kW per rack — the optical cabling gets unmanageable at higher densities.

What inlet temperature should I design for?

ASHRAE A4 (45 °C) is the new baseline for AI halls. Some hyperscalers run A3 (40 °C) with derated switch throughput. Avoid A1/A2 (≤27 °C) designs for new AI builds — they waste energy on overcooling and constrain DLC efficiency because the coolant delta-T shrinks.

How much does DLC add to a GPU server’s capex?

Typical cold-plate DLC adds 8–15% to the server’s capex (manifold, hoses, quick-disconnects, CDU share). Two-phase immersion adds 20–30% but eliminates the need for server fans and most rack-level cooling infrastructure. TCO breakeven with air cooling is usually 18–30 months at densities above 40 kW/rack.

8. Conclusion: A Network-Aware Cooling Strategy

Liquid cooling is no longer optional for new AI factories, but it is also not a stand-alone project. The ToR switch, the DPU, the optical transceiver, the rack airflow, and the CDU must be designed together. Buyers who treat cooling as “facilities” and networking as “compute” end up with stranded capacity and thermal bottlenecks at exactly the moments when collective-comms traffic peaks.

For data centers in the 40–80 kW range, direct liquid cooling with a row-level CDU and air-cooled ToR switches is the most flexible, lowest-risk path. Above 100 kW, immersion becomes attractive but forces the network into a separate physical layout. In either case, choose ASHRAE A4-rated switches, port-side intake airflow, and DPU-aware cold-plate coverage — and verify every component against your specific coolant inlet temperature.

For data centers in the 40–80 kW range, direct liquid cooling with a row-level CDU and air-cooled ToR switches is the most flexible, lowest-risk path. Above 100 kW, immersion becomes attractive but forces the network into a separate physical layout. In either case, choose ASHRAE A4-rated switches, port-side intake airflow, and DPU-aware cold-plate coverage — and verify every component against your specific coolant inlet temperature.

9. Heat Reuse and CDU Topology Choices

One under-discussed benefit of liquid cooling is that the heat it captures is useful. Facility warm water leaving a CDU typically runs 50–65 °C — hot enough for:

  • District heating networks — European operators (e.g., Equinix Paris, DataCenter Donauworth) pipe waste heat to nearby residential and commercial users, recovering 60–80% of the input energy.
  • On-site absorption chillers — for mixed-use campuses where office buildings still need chilled water, the GPU heat can drive an absorption chiller at surprisingly good COP (0.6–0.8) compared to electric chillers (COP 4–6 but with grid losses).
  • Agricultural and aquaculture — fish farms, greenhouse heating, and algae bioreactors can absorb low-grade heat that would otherwise be wasted.

From a CDU topology standpoint, three patterns dominate 2026 deployments:

Topology Heat Reuse Potential Network Rack Constraints Best Fit
Single-loop direct (no CDU) Low (facility-water only) None Pilot / under 200 kW total
Two-loop with row CDU Medium (warm water 40–55 °C) Top-of-rack plumbing restricts 1U ToR airflow Most common AI factories, 1–10 MW
Three-loop with heat exchanger + facility CDU High (district-heating-grade 55–70 °C) Plumbing + heat exchanger room eats ~10% of floor space Sustainability-focused builds, 5 MW+

The network impact of topology choice is small but real: in a three-loop design, the secondary CDU and heat exchanger must sit outside the hot aisle, and the facility-water loop must run with sufficient pressure to feed the district heating network. This affects where you can place spine switches and aggregation routers — typically in a dedicated network-only row that is air-cooled independently from the GPU rows.

For data center operators looking to refresh their fabric without a full capex cycle, used ConnectX adapters, Quantum switches, and DAC cables from a trusted supplier can deploy cleanly into DLC racks when properly retested.

Explore related categories at Zhongxin Data

Need help sizing a CDU, validating switch thermal specs, or sourcing DLC-ready hardware that has been retested? Contact Zhongxin Data on WhatsApp +86 13068616914 or browse our full data center catalog.