Introduction: When the Fabric Breaks, the Cluster Stops

In modern AI data centers, the network fabric is no longer just plumbing—it is the critical infrastructure that determines whether a GPU cluster delivers exaFLOPS or sits idle. A single flapping port on an NVIDIA SN5600 switch can cascade into a collective operation timeout that stalls an entire training run costing thousands of dollars per hour. As fabrics scale to tens of thousands of endpoints at 400G and 800G, the traditional “ping and traceroute” approach to network troubleshooting is woefully inadequate.

This field guide distills the troubleshooting methodologies used by data center network engineers operating large-scale InfiniBand and Ethernet fabrics. We cover the diagnostic toolchains, common failure modes, and systematic approaches to isolating faults in production AI clusters. Whether you operate a small HPC cluster with Mellanox ConnectX-7 adapters or a hyperscale AI training fabric with NVIDIA Quantum-2 InfiniBand switches, these techniques will help you reduce mean-time-to-resolution (MTTR) and keep your GPUs fed with data.

Key Insight: In AI training fabrics, a 0.1% packet loss on a single rail can reduce collective operation throughput by 40% or more. Silent degradation is the enemy—your monitoring must catch problems before they manifest as slow jobs.

The Troubleshooting Toolkit: Layer by Layer

Effective network troubleshooting in data center fabrics requires tools at every layer of the stack. The table below maps common symptoms to the diagnostic tools and the OSI layer where the fault typically resides.

Symptom Likely Layer Primary Diagnostic Tools Typical Root Cause
Port down / link not established L1 (Physical) mellanox_burn, ethtool, mlxlink, optical power meter Dirty fiber, damaged transceiver, cable bend radius, module compatibility
Intermittent CRC errors L1 (Physical) ethtool -S, mlxlink, BER counters Marginal optics, connector contamination, EMI, temperature drift
High latency on specific path L2/L3 (Data/Network) perfquery (IB), show counters, ibdump, RoCE counters Congestion hotspot, ECMP hash polarization, PFC deadlock
Collective operation timeout L4/Transport NCCL debug logs, ib_write_bw, osu_benchmark, SHARP telemetry SHARP aggregation node failure, adaptive routing misconfiguration, silent packet drop
Training job hangs (all-reduce stall) L7/Application NCCL NCCL_DEBUG=INFO, nsys profiler, DCGM Rank imbalance, deadlocked dependency, NCCL topology mismatch

InfiniBand-Specific Tools

The InfiniBand ecosystem provides a rich set of diagnostic tools that have no direct Ethernet equivalent. Understanding these tools is essential for troubleshooting Quantum-2 and Quantum-3 based fabrics:

  • ibstat / ibv_devinfo: Verify port state (Active/Down/Initializing), link width, and speed. A port showing State: Active but Physical State: Polling indicates a link negotiation problem.
  • perfquery: Query performance counters on IB ports. Key counters include SymbolErrors, LinkErrorRecovery, PortXmitDiscards, PortRcvConstraintErrors. A rising LinkErrorRecovery counter signals marginal optics.
  • ibnetdiscover: Maps the entire fabric topology, showing every switch and endpoint with their LIDs. Critical for verifying cabling correctness and detecting topology mismatches after hardware swaps.
  • ibdiagnet: NVIDIA’s comprehensive fabric diagnostic tool that runs automated health checks across the entire subnet. Reports link quality, congestion, routing anomalies, and SM configuration issues.
  • mlxlink: Mellanox-specific tool for checking physical layer parameters—optical signal-to-noise ratio, pre-FEC BER, post-FEC BER, and eye diagram quality. Essential for diagnosing marginal transceivers.
  • saquery: Queries the Subnet Administrator for routing tables, path records, and multicast group information. Use it to verify that traffic is taking expected paths.

Ethernet / RoCE Tools

For Ethernet fabrics running RoCE v2 (used in NVIDIA Spectrum-X and Ultra Ethernet Consortium architectures), the toolset overlaps with traditional Ethernet diagnostics but adds RoCE-specific counters:

  • ethtool -S: Read NIC and switch port counters. Key RoCE counters: rx_roce_cnp (CNP packets), tx_roce_cnp, rx_roce_drops, rx_roce_pps. CNP (Congestion Notification Packets) count indicates active congestion control.
  • show tech-support: Dumps all switch counters, logs, and configuration in one command. NVIDIA Cumulus Linux and ONiX provide this for rapid escalation to vendor support.
  • mellanox_burn / cable diagnostics: Time Domain Reflectometry (TDR) on DAC cables to locate cable faults to the meter. Also checks transceiver EEPROM for authenticity and operating parameters.
  • PFC watchdog counters: show pfcwd stats reveals if PFC watchdog is disabling ports due to stuck queues—a common symptom of head-of-line blocking.

Common Failure Modes and Resolution Playbooks

1. Port Flapping: The Silent Killer

Port flapping—a link repeatedly going down and up—is one of the most destructive failure modes in AI fabrics. In an all-reduce operation, every GPU must participate; a single flapping port causes the collective to stall repeatedly, and NCCL will eventually timeout the entire training job.

Diagnostic steps:

  1. Check ethtool -S or perfquery for LinkDowned counter. If incrementing, the port is genuinely going down.
  2. Run mlxlink to check BER. A pre-FEC BER above 1E-4 on a 400G link suggests marginal optics or dirty connectors.
  3. Check transceiver temperature. NVIDIA SN5600 switches report per-port optical module temperature; values above 70°C indicate thermal issues.
  4. Inspect fiber connectors with a fiber inspection scope. Even invisible contamination can cause intermittent CRC errors at 400G.
  5. Verify DAC cable length and specifications. Active DACs beyond their rated length exhibit temperature-dependent flapping.
BER Range Link Health Recommended Action Expected MTTR
< 1E-12 Healthy No action needed. Monitor periodically. N/A
1E-12 to 1E-8 Marginal Clean connectors, check temperature, schedule replacement 30 min
1E-8 to 1E-4 Degraded Replace transceiver/cable immediately, reduce link speed if possible 15 min
> 1E-4 Critical / Flapping Port will likely go down. Isolate, replace optics, re-test 45 min

2. Congestion Hotspots and PFC Deadlock

In lossless Ethernet fabrics using PFC (Priority Flow Control) and ECN (Explicit Congestion Notification), a congestion hotspot can cascade into a PFC deadlock—a situation where ports on multiple switches are paused waiting for each other, creating a permanent traffic stall.

Symptoms:

  • Throughput drops to near-zero on multiple flows simultaneously
  • PFC pause frame counters spike (tx_pause_storms, rx_pause_frames)
  • NCCL reports timeout on all-reduce but individual ib_write_bw tests pass in isolation
  • PFC watchdog may or may not trigger depending on configuration

Resolution steps:

  1. Identify the stalled queue: Run show counters interface on each switch in the path. Look for tx_discards or egress_drops on a specific priority (usually priority 3 for RoCE).
  2. Trace the PFC storm: Use show pfcwd stats to find which ports are receiving PFC pause frames. The source of the pause is downstream of the congestion point.
  3. Check ECN configuration: Verify that ECN marking thresholds are set correctly. If ECN is too aggressive, CNP storms can cause throughput collapse. If too lenient, PFC triggers before congestion control reacts.
  4. Verify ECMP hash distribution: Use show ecmp to check if flows are polarizing onto a single link. This is common with fat-tree topologies when the hash seed is poorly configured.
  5. Check for deadlock recovery: NVIDIA Spectrum switches support automatic deadlock detection and recovery (port reset). Verify this is enabled: show pfcwd config.

Pro Tip: In NVIDIA Spectrum-X fabrics, the BlueField-3 DPU can perform per-flow congestion telemetry. Enabling telemetry streaming on the DPU gives you per-flow RTT and queue depth without affecting data plane performance.

3. InfiniBand SHARP Aggregation Failures

NVIDIA’s SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) offloads collective operations to the network switches. When a SHARP aggregation node fails, collective operations fall back to software-based all-reduce, causing a 2-5x performance degradation that is difficult to diagnose without understanding the SHARP architecture.

Symptoms:

  • NCCL reports all-reduce completion but latency is 2-5x higher than baseline
  • ib_write_bw point-to-point tests show normal performance
  • SHARP group creation fails or SHARP trees show incomplete membership
  • Job runs succeed but take significantly longer than expected

Diagnostic steps:

  1. Verify SHARP is enabled: Check NCCL environment variables: NCCL_IB_HCA=mlx5, NCCL_NET_GDR_LEVEL=PHB, NCCL_DEBUG=INFO. Look for “SHARP” in the NCCL log output.
  2. Check SHARP manager: The SHARP aggregation manager runs on the subnet manager (UFM/SM). Query sharp_am_get_info to verify the aggregation manager is running and has assigned trees.
  3. Inspect SHARP trees: Use sharp_coll_dump to dump the current SHARP tree topology. A broken tree (missing nodes) indicates a switch or endpoint that lost SHARP membership.
  4. Check switch SHARP capability: Not all switches support SHARP. Verify that the switches designated as aggregation nodes (usually spine switches) have SHARP-capable hardware (Quantum-2 SN5600 and later).
  5. Review UFM events: NVIDIA UFM logs SHARP-related events including tree rebuilds, aggregation node failures, and membership changes. Check the UFM event log for anomalies during the problem window.

4. Silent Performance Degradation: The Hardest Problem

The most insidious fabric problems are those where the network appears healthy—links are up, error counters are zero, point-to-point bandwidth tests pass—but collective operations are slower than expected. This “silent degradation” is common in large fabrics and requires correlation of multiple telemetry sources to diagnose.

Common causes of silent degradation:

Root Cause Detection Method Performance Impact Resolution
Adaptive routing misconfiguration Compare per-path latency with osu_latency; check show adaptive-routing on switches 10-30% collective throughput loss Reconfigure AR packet spraying; verify AR-capable switches
ECMP hash polarization Per-port throughput analysis shows uneven distribution (>20% variance) 15-40% effective bandwidth loss Adjust ECMP hash seed; enable per-flow entropy
GPU Direct RDMA (GDR) fallback Check NCCL log for “GDR” vs “bounce buffer”; nvidia-smi P2P topology 30-60% all-reduce throughput loss Verify PCIe topology, NCCL_NET_GDR_LEVEL, BAR1 size
NUMA imbalance numactl --hardware, lspci NUMA affinity, NCCL topology XML 10-25% cross-NUMA penalty Pin processes to correct NUMA node; use NCCL_IB_QPS_PER_CONNECTION
Firmware/driver version mismatch ibv_devinfo -v, mst version, compare across nodes Variable; can cause AR failures or GDR issues Standardize on tested firmware/driver combinations

Building a Proactive Monitoring Stack

The best troubleshooting strategy is preventing problems before they impact workloads. A modern data center fabric monitoring stack should include the following components:

  1. UFM (Unified Fabric Manager): NVIDIA’s fabric management platform for InfiniBand. Provides real-time topology visualization, performance monitoring, alerting, and automated remediation. Essential for Quantum-2/Quantum-3 fabrics.
  2. Cumulus Linux NetQ: Telemetry and observability for Ethernet fabrics running on NVIDIA Spectrum switches. Provides streaming telemetry, anomaly detection, and historical analysis.
  3. Prometheus + Grafana: Open-source monitoring stack. Collect switch and NIC counters via SNMP exporters or streaming telemetry (gNMI). Build dashboards for key metrics: BER, PFC pause frames, ECN marking rates, per-port throughput.
  4. NCCL Telemetry: NCCL can export per-collective-operation timing data. Correlating NCCL timing spikes with switch counter anomalies is the most effective way to catch silent degradation.
  5. DCGM (Data Center GPU Manager): While primarily a GPU monitoring tool, DCGM can detect GPU Direct RDMA failures and PCIe topology issues that manifest as network problems.

Escalation Matrix: When to Call Vendor Support

Not every problem can be solved with open-source tools. Knowing when to escalate to NVIDIA or your hardware vendor is critical for minimizing downtime. Use the following matrix as a guideline:

Scenario Self-Service Resolution Time Escalate When Support Tier
Port down, known cause (cable, transceiver) < 30 min Multiple ports down simultaneously, no common cause Tier 2 (switch vendor)
Congestion hotspot, single flow 1-2 hours PFC deadlock persists after reconfiguration Tier 2 (fabric specialist)
SHARP tree failures 2-4 hours SHARP manager crashes or trees cannot be rebuilt Tier 3 (NVIDIA UFM team)
Silent degradation, unclear cause 4-8 hours After exhausting diagnostic tools, correlated telemetry reviewed Tier 3 (vendor fabric engineering)

Best Practices for Reducing MTTR

Based on field experience operating large-scale AI fabrics, the following practices consistently reduce mean-time-to-resolution:

  • Maintain a known-good baseline: Run osu_benchmark and ib_write_bw across all node pairs when the fabric is healthy. When problems arise, compare against this baseline to immediately identify degraded paths.
  • Automate fabric health checks: Script daily runs of ibdiagnet or show tech-support with diff-based alerting. A change in counter values is more actionable than absolute thresholds.
  • Standardize firmware and driver versions: Version mismatches between switches, NICs, and the subnet manager are a leading cause of subtle interoperability issues. Maintain a tested compatibility matrix and enforce it during deployment.
  • Label and document cabling: In fabrics with thousands of cables, unlabeled connections turn a 15-minute cable swap into a 2-hour hunt. Use consistent labeling schemes and maintain a digital cable map.
  • Keep spare optics and cables on hand: A transceiver failure at 3 AM should not wait for procurement. Maintain a spares inventory proportional to fabric size (typically 5-10% of deployed optics).
  • Correlate NCCL logs with fabric telemetry: NCCL’s NCCL_DEBUG=INFO output contains timestamps and rank information that, when correlated with switch counter data, pinpoint the exact moment and location of problems.

Conclusion

Data center network troubleshooting for AI fabrics is fundamentally different from traditional enterprise networking. The scale, speed (400G/800G), and sensitivity of collective operations mean that problems invisible to traditional monitoring can cause catastrophic performance degradation. By building a layered diagnostic toolkit, understanding the unique failure modes of InfiniBand and RoCE fabrics, and implementing proactive monitoring, operators can keep their GPU clusters running at peak efficiency.

At Zhongxin Data, we supply the data center hardware that powers these fabrics—from NVIDIA Mellanox InfiniBand adapters and Quantum switches to Broadcom MegaRAID controllers and enterprise SSDs. Whether you are building a new AI training cluster or maintaining an existing fabric, our inventory of genuine and used data center components ensures you have the right hardware at the right price.

Explore our product categories:

Frequently Asked Questions

What is the most common cause of port flapping in 400G InfiniBand fabrics?

The most common cause is dirty or contaminated fiber optic connectors. At 400G speeds, even microscopic contamination on a connector end-face can cause bit errors that exceed the FEC correction capability, triggering link recovery. The second most common cause is marginal transceivers operating outside their temperature specification. Always use a fiber inspection scope before mating connectors and monitor transceiver temperatures via mlxlink.

How do I diagnose PFC deadlock vs. normal congestion in a RoCE fabric?

PFC deadlock is characterized by multiple ports simultaneously stuck in a paused state with no traffic flowing, while normal congestion shows periodic pauses with throughput still progressing. Use show pfcwd stats to check if the PFC watchdog has detected and recovered from deadlock events. If deadlock occurs repeatedly, verify your topology has no cyclic dependencies on the RoCE priority class and that deadlock detection/recovery is enabled on all switches.

Why does my NCCL all-reduce run slower than point-to-point bandwidth tests suggest?

Point-to-point tests (like ib_write_bw) only stress a single path, while NCCL all-reduce exercises the entire fabric topology simultaneously. Common causes include: (1) SHARP offload not being used (check NCCL logs for SHARP activation), (2) adaptive routing not enabled or misconfigured, (3) ECMP hash polarization concentrating flows onto fewer paths, and (4) GPU Direct RDMA not active (check for “bounce buffer” in NCCL logs). Run osu_benchmark suite for a more representative collective operation test.

What BER threshold should trigger proactive cable replacement?

For 400G and 800G links, a pre-FEC BER above 1E-8 warrants proactive replacement. At this level, the link is still operational (FEC corrects errors), but it indicates marginal hardware that will likely fail soon. A BER above 1E-4 typically causes port flapping and should be treated as an urgent issue. Use mlxlink to monitor BER trends—a rising BER over days or weeks is a stronger signal than a single measurement.

How often should I run fabric diagnostics like ibdiagnet?

For production AI fabrics, run ibdiagnet at least weekly as a scheduled health check, and after any hardware change (switch replacement, cable swap, firmware update). For critical training runs, run a quick diagnostic before starting the job. NVIDIA UFM can automate this with scheduled scans and alerting on anomalies. For Ethernet fabrics, equivalent checks via show tech-support should be run on the same cadence.