Author: Zhongxin Data | Topic: Data Center Storage Networking
Modern AI training clusters, real-time analytics, and high-frequency trading platforms are no longer bottlenecked by CPU or GPU alone—they are increasingly constrained by how fast storage can feed data into compute. Traditional SCSI-based SAN protocols such as iSCSI and Fibre Channel (FC) were designed for spinning disks and millisecond latencies. With all-flash arrays now delivering microsecond-level access times, the storage network itself has become the limiting factor. NVMe over Fabrics (NVMe-oF) solves this by extending the high-performance NVMe command set across a network fabric, allowing remote SSDs to behave like local drives.
In this guide, Zhongxin Data breaks down NVMe-oF transport protocols, compares NVMe/RoCE v2, NVMe/TCP, and NVMe/FC, and explains how to architect a low-latency storage fabric for AI clusters, VMware vSphere environments, and cloud-native Kubernetes deployments.
“NVMe-oF is not simply a faster SAN—it removes the SCSI translation layer entirely, enabling storage latency to approach that of direct-attached NVMe SSDs while preserving shared access and scale-out resilience.”
What Is NVMe-oF and Why Does It Matter?
NVMe over Fabrics (NVMe-oF) is a protocol specification defined by NVM Express that allows NVMe commands and data to traverse a network between a host and a target SSD or flash array. Unlike iSCSI, which wraps SCSI commands inside IP packets, NVMe-oF carries the lightweight NVMe queue-pair model end to end. The result is substantially lower latency, higher IOPS per CPU core, and simpler command processing.
The primary use cases for NVMe-oF include:
- AI/ML training clusters: Feeding GPUs with massive datasets from shared NVMe storage without local copy delays.
- High-frequency trading and OLTP databases: Reducing transaction latency from hundreds of microseconds to tens of microseconds.
- Virtualized enterprise workloads: Enabling vMotion and live migration across hosts with shared all-flash storage.
- Kubernetes and container platforms: Delivering persistent volumes with bare-metal NVMe performance.
- Media and entertainment rendering: Sharing 8K/12K footage across multiple render nodes with frame-accurate playback.
Because NVMe-oF preserves the NVMe command set, operating systems and applications require only a standard NVMe-oF initiator, making integration far simpler than proprietary clustered file systems.
NVMe-oF Transport Protocols Compared
NVMe-oF can run over several transports. The three most common in enterprise and hyperscale deployments are RDMA over Converged Ethernet (RoCE v2), TCP/IP, and Fibre Channel (NVMe/FC). Each has distinct latency, hardware, distance, and management characteristics.
| Transport | Latency | Network Hardware | Distance | Best For |
|---|---|---|---|---|
| NVMe/RoCE v2 | ~2–10 μs | RDMA-capable NIC (ConnectX-6/7, Broadcom 57414) | Single datacenter / L3 routed | AI clusters, HPC, ultra-low latency databases |
| NVMe/TCP | ~20–50 μs | Standard Ethernet NIC, no RDMA required | Unlimited IP reachability | General enterprise, cloud, VM workloads |
| NVMe/FC | ~10–25 μs | 16/32/64 Gb FC HBAs and FC switches | Campus / metro | Existing FC SAN investments, regulated enterprises |
RoCE v2 delivers the lowest latency because it bypasses the kernel TCP/IP stack and uses RDMA to move data directly between application memory on the host and the SSD controller. However, it requires lossless Ethernet, Priority Flow Control (PFC), and explicit congestion management to avoid packet drops that would degrade RDMA performance. NVMe/TCP trades a small latency penalty for enormous operational simplicity: it works on any standard Ethernet switch and can be routed across datacenters. NVMe/FC leverages existing Fibre Channel infrastructure and offers deterministic performance, making it attractive for organizations already invested in Brocade or Cisco MDS SANs.
Performance Comparison: IOPS, Latency, and Bandwidth
The following table summarizes realistic performance expectations for a single modern NVMe-oF target array connected through different transports. Actual results depend on SSD model, queue depth, CPU count, and network tuning.
| Metric | NVMe/RoCE v2 | NVMe/TCP | NVMe/FC (32 Gb) |
|---|---|---|---|
| Typical read latency | 10–25 μs | 30–80 μs | 25–50 μs |
| 4 KB random read IOPS (per array) | 3–8 million | 2–5 million | 1.5–3 million |
| Sequential read bandwidth | Up to 400 Gb/s | Up to 200 Gb/s | Up to 128 Gb/s aggregate |
| CPU overhead per IOPS | Very low | Moderate | Low |
| Switch requirements | Lossless Ethernet, PFC/ECN | Standard Layer-2/3 Ethernet | Fibre Channel switches only |
For workloads where every microsecond counts—such as checkpoint loading in large language model (LLM) training—NVMe/RoCE v2 is the clear winner. For general-purpose virtualization and cloud storage, NVMe/TCP provides a compelling balance of performance and manageability. NVMe/FC remains relevant where organizations value a dedicated, isolated storage fabric and have existing operational expertise.
Network Topology for NVMe-oF
NVMe-oF can be deployed in several topologies depending on scale, performance target, and budget:
- Direct-attach (point-to-point): A host connects directly to an NVMe-oF target over a single cable. Simplest but offers no redundancy.
- Single switch: Multiple hosts and targets connect to one high-radix Ethernet or FC switch. Suitable for small clusters and proof-of-concept deployments.
- Leaf-spine fabric: Hosts attach to leaf switches, which uplink to spine switches. This is the standard design for AI clusters and scale-out storage, providing predictable latency and non-blocking bandwidth.
- Collapsed core: For mid-size enterprises, a pair of 100G/400G switches can serve as both leaf and spine, reducing cost while preserving redundancy.
When deploying NVMe/RoCE v2, network architects must enable:
- Priority Flow Control (PFC): Pauses traffic on the lossless priority to prevent buffer overflow during congestion.
- Explicit Congestion Notification (ECN): Signals congestion to endpoints so they throttle before packet loss occurs.
- Data Center Bridging Exchange (DCBX): Automatically negotiates PFC and ECN parameters between NICs and switches.
- Jumbo frames (MTU 9000): Reduces header overhead and improves throughput for large sequential transfers.
Without these features, RoCE v2 can suffer from severe performance degradation under heavy load. By contrast, NVMe/TCP tolerates standard IP networking but should still run on low-latency switches with sufficient buffer depth.
Hardware Requirements: NICs, Switches, and Cables
Building an NVMe-oF fabric requires carefully matched components. The following table lists typical hardware options for each transport at 2026 price and performance levels.
| Component | NVMe/RoCE v2 | NVMe/TCP | NVMe/FC |
|---|---|---|---|
| Host adapter | NVIDIA Mellanox ConnectX-6 DX/ConnectX-7 | Standard 25/100 GbE NIC | Emulex/Broadcom 32 Gb FC HBA |
| Switch example | NVIDIA Spectrum SN3700/SN4700, Dell Z9264F-ON | Any 25/100/400 GbE switch | Brocade G630/G730, Cisco MDS 9132T |
| Cabling | DAC, AOC, or single-mode transceivers | Standard Ethernet copper/fiber | OM4/OM5 multimode or single-mode fiber |
| Typical port speed | 100 GbE / 200 GbE / 400 GbE | 25 GbE / 100 GbE / 400 GbE | 32 Gb FC / 64 Gb FC |
Zhongxin Data supplies enterprise-grade NVIDIA Mellanox ConnectX InfiniBand and Ethernet adapters, Spectrum SN series switches, and Broadcom MegaRAID tri-mode controllers that integrate seamlessly with NVMe-oF storage targets. Our hardware program lets organizations deploy high-speed storage networks at a fraction of OEM list prices.
NVMe-oF in AI and HPC Clusters
AI training jobs, especially large language models and computer vision at scale, read training data and write checkpoints to storage at rates that overwhelm traditional NAS or SAN architectures. A single DGX H100 node can saturate multiple 200 Gb/s links during checkpoint operations. NVMe-oF, particularly over RoCE v2 or InfiniBand, provides the bandwidth and low latency required to keep GPUs fed.
Key design considerations for AI storage fabrics include:
- Parallel filesystem integration: GPFS, Lustre, WekaFS, and BeeGFS can use NVMe-oF targets as the underlying block devices, exposing a shared namespace to thousands of compute nodes.
- Balanced fan-out: Each storage target should be connected at the same total bandwidth as the compute clients to avoid hotspots.
- Checksum and T10-DIF: End-to-end data integrity protects against silent corruption during high-speed transfers.
- Multipath and failover: Hosts should use multiple ports and switches so a single link failure does not stall training.
Organizations deploying NVIDIA Quantum InfiniBand fabrics can also leverage native NVMe-oF support through the InfiniBand RDMA transport, achieving even lower latency than RoCE v2 in tightly coupled HPC environments.
Management and Security Best Practices
Despite its performance benefits, NVMe-oF introduces new management requirements. Administrators should implement:
- Zoning and VLAN isolation: Separate storage traffic from general datacenter traffic to improve security and predictability.
- TLS for NVMe/TCP: Encrypt data in transit when compliance mandates confidentiality.
- CHAP or DH-HMAC authentication: Verify initiator identity before allowing access to storage namespaces.
- Monitoring queue depth and latency: Use vendor tools or open-source collectors like Prometheus and Telegraf to track per-port performance.
- Regular firmware updates: NIC, HBA, switch, and SSD firmware should be kept aligned to avoid interoperability issues.
For regulated industries, NVMe/FC remains attractive because it runs on a physically isolated SAN, simplifying audit and compliance workflows.
Cost-Benefit Analysis: When to Choose Each Transport
Selecting the right NVMe-oF transport depends on workload sensitivity, existing infrastructure, and operational expertise. The table below summarizes a practical decision framework.
| If Your Priority Is… | Choose | Reason |
|---|---|---|
| Lowest possible latency | NVMe/RoCE v2 | Kernel bypass and RDMA deliver sub-10 μs network overhead. |
| Lowest infrastructure cost | NVMe/TCP | Uses standard NICs and switches; no RDMA tuning required. |
| Long-distance replication | NVMe/TCP | Routable IP allows metro and inter-datacenter links. |
| Existing FC SAN | NVMe/FC | Leverages existing HBAs, switches, cabling, and staff skills. |
| AI/HPC at scale | NVMe/RoCE v2 or InfiniBand | Highest bandwidth density and lowest collective latency. |
Many enterprises start with NVMe/TCP for general workloads and add NVMe/RoCE v2 islands for performance-critical applications such as AI training or real-time databases.
NVMe-oF Deployment Checklist
Before putting NVMe-oF into production, verify each layer of the stack. A single misconfigured switch or outdated firmware version can turn a sub-10 μs fabric into an unstable bottleneck. Use the checklist below as a pre-flight guide:
- Firmware alignment: Confirm NIC/HBA, switch, SSD, and storage-controller firmware are on mutually supported versions. Vendor interoperability matrices are essential.
- Lossless Ethernet for RoCE: Enable PFC on the storage priority queue, configure ECN thresholds, and validate DCBX negotiation end to end.
- MTU consistency: Set host, switch, and target interfaces to MTU 9000 for jumbo frames. Mismatched MTU is a common cause of poor throughput.
- Queue depth tuning: Match host queue depths to target capabilities. Too shallow limits IOPS; too deep inflates latency under congestion.
- Multipath configuration: Configure active-active or active-passive multipath with a policy appropriate to your OS and switch topology.
- Subnet and VLAN planning: Keep storage traffic in isolated broadcast domains or VRFs. Avoid routing storage traffic through general-purpose firewalls.
- Authentication: Enable CHAP, DH-HMAC, or TLS where supported to prevent unpermitted namespace access.
- Monitoring baseline: Capture latency, IOPS, bandwidth, and error counters before go-live. Establish alerting thresholds for retransmits and PFC pause frames.
- Failover testing: Physically disconnect one path and verify multipath failover completes within the SLA.
- Backup and recovery: Ensure existing backup software supports NVMe-oF targets and can restore at the required RTO/RPO.
Running a staged proof of concept with representative workloads—rather than synthetic benchmarks alone—will surface real-world behavior such as congestion under checkpoint bursts, multipath asymmetry, and operating-system driver quirks.
Common Pitfalls and How to Avoid Them
Even experienced storage teams can stumble when transitioning from FC or iSCSI to NVMe-oF. The most frequent issues include:
- Assuming lossy Ethernet works for RoCE: Without PFC and ECN, even brief congestion triggers RDMA retransmits that collapse throughput. Always configure a lossless priority class.
- Undersizing switch buffers: Bursty AI checkpoint traffic can overwhelm shallow-buffer switches. Choose switches with deep packet buffers or dynamic buffer allocation.
- Neglecting CPU affinity: For NVMe/TCP, distribute interrupts across NUMA-local cores to prevent a single CPU from becoming the bottleneck.
- Mismatched driver versions: Initiator and target drivers from different release trains can cause command timeouts or namespace discovery failures.
- Ignoring cable quality: High-speed 100G/400G links are sensitive to dust, bend radius, and signal integrity. Use certified optics and clean connectors before installation.
By treating NVMe-oF as a tightly coupled compute-storage network rather than a generic IP extension, teams can avoid these pitfalls and achieve predictable performance.
Conclusion
NVMe-oF has become the foundation of modern high-performance storage networking. By extending the NVMe queue-pair model across RoCE v2, TCP, or Fibre Channel, it enables shared flash storage to deliver near-local performance. For AI clusters, the choice is usually NVMe/RoCE v2 or native InfiniBand. For mainstream enterprise virtualization, NVMe/TCP offers an attractive balance of simplicity and speed. For organizations with existing Fibre Channel investments, NVMe/FC provides a smooth upgrade path without forklift replacement.
At Zhongxin Data, we help businesses design, source, and deploy cost-optimized NVMe-oF fabrics using used NVIDIA Mellanox ConnectX adapters, Spectrum switches, and Broadcom RAID controllers. Whether you are building a small all-flash SAN or a multi-rack AI training cluster, our team can recommend the right transport, topology, and hardware mix to meet your latency and budget targets.
Related Products
- NVIDIA Mellanox ConnectX Ethernet & InfiniBand Adapters
- Data Center Switches: 100G, 400G, and 800G
- Broadcom MegaRAID & HBA Storage Controllers
- DDR5 RDIMM Server Memory
Frequently Asked Questions
What is the main advantage of NVMe-oF over iSCSI?
NVMe-oF carries the lightweight NVMe command set directly over the network, eliminating SCSI translation overhead and enabling latencies in the tens of microseconds. iSCSI wraps SCSI commands in IP packets and typically adds hundreds of microseconds of latency, making it unsuitable for all-flash workloads that demand microsecond response times.
Can I run NVMe-oF over my existing Ethernet switches?
Yes, if you use NVMe/TCP. It runs on standard Ethernet NICs and switches without RDMA support. However, if you choose NVMe/RoCE v2, your switches must support lossless Ethernet features such as Priority Flow Control (PFC), Explicit Congestion Notification (ECN), and Data Center Bridging Exchange (DCBX).
Is NVMe-oF only for large hyperscale datacenters?
No. NVMe-oF is used in environments ranging from small all-flash SANs with two hosts to multi-rack AI training clusters. NVMe/TCP, in particular, lowers the barrier to entry because it does not require specialized RDMA hardware or advanced switch configuration.
What operating systems support NVMe-oF?
Modern Linux kernels include NVMe-oF initiator and target support for TCP, RDMA, and FC. Windows Server supports NVMe-oF through inbox or vendor drivers. VMware vSphere supports NVMe/TCP and NVMe/FC for NVMe-oF datastores in recent releases.
How does NVMe-oF relate to InfiniBand?
InfiniBand natively supports RDMA and can carry NVMe-oF traffic with extremely low latency. Many top-tier HPC and AI clusters use InfiniBand rather than Ethernet/RoCE because it provides integrated congestion management, higher bandwidth density, and mature support for MPI collective operations.