The industry is currently obsessed with TFLOPS. With the unveiling of NVIDIA’s Blackwell B200 and the liquid-cooled GB200 NVL72 racks, the numbers are staggering: 20 petaflops of FP4 compute per GPU, 1.8TB/s of NVLink bandwidth, and a 72-GPU domain that acts as a single massive compute unit.
But here is the engineering reality that keeps infrastructure architects up at night: Compute is easy; communication is hard.
When you scale a Blackwell cluster beyond a single rack into the tens of thousands of GPUs required for frontier model training, the “Networking Wall” becomes the primary antagonist. At this scale, the traditional methods of moving data across the data center fall apart. We are no longer just sending packets; we are orchestrating a synchronous, multi-tier symphony of data movement where a single microsecond of tail latency can stall a $100M training run.
In this deep dive, we’re going to explore how we optimize RDMA over Converged Ethernet (RoCE) for multi-tier collective communications in Blackwell clusters. We’ll look at why the shift to Ethernet is happening, the intricacies of rail-optimized topologies, and the low-level tuning required to make a “lossy” fabric behave like a lossless one.
The Blackwell Context: Why the Network Boundary Shifted
To understand the networking optimization, we first have to understand the physical shift in Blackwell. In the Hopper generation (H100), the NVLink domain was typically limited to 8 GPUs within a single server. Anything beyond 8 GPUs required hitting the InfiniBand or RoCE network.
With Blackwell GB200 NVL72, the boundary has moved. We now have 72 GPUs connected via a massive copper backplane (NVLink Switch System), providing a staggering 130TB/s of aggregate bandwidth.
The catch? Once you leave that 72-GPU rack to talk to the other 1,000 racks in your cluster, you are transitioning from the “perfect” world of NVLink to the “challenging” world of the leaf-spine fabric. This transition creates a multi-tier communication hierarchy:
- Tier 1: Intra-Rack (NVLink): Ultra-high bandwidth, near-zero latency, hardware-managed.
- Tier 2: Inter-Rack (RoCEv2): High bandwidth, managed via NICs (like the ConnectX-8 or BlueField-3), sensitive to congestion and topology.
Our job is to ensure that the handoff between Tier 1 and Tier 2 is seamless.
Why RoCE? The Case for Ethernet in AI
For a long time, InfiniBand was the undisputed king of AI networking because of its native lossless nature and credit-based flow control. However, the tide is shifting toward RoCEv2 (RDMA over Converged Ethernet) for three reasons:
- Economies of Scale: Ethernet is the lingua franca of the data center. Hyperscalers can leverage their existing supply chains and operational expertise.
- The Ultra Ethernet Consortium (UEC): Massive industry backing is rapidly evolving Ethernet to handle the specific “incast” patterns of AI workloads.
- Bandwidth Parity: With 400G and 800G Ethernet switches now mainstream, the raw throughput gap between IB and Ethernet has vanished.
But RoCE is “RDMA over UDP.” It’s fundamentally built on a lossy foundation. To make it work for Blackwell-scale collectives (All-Reduce, All-to-All), we have to engineer the “lossless” behavior back into the stack.
Rail-Optimized Topology: The Blueprint for Scale
In a traditional data center, you use a Fat-Tree or Clos topology to provide any-to-any connectivity. In a Blackwell cluster, we use a Rail-Optimized design.
In a rail-optimized cluster, we ensure that “GPU 0” in every rack is connected to the same leaf switch (the “GPU 0 Rail”). When a collective operation like All-Reduce happens, the GPUs talk to their peers in the same “rail” across racks.
Why this matters for Blackwell:
Blackwell GPUs are often deployed in “SuperNIC” configurations. By aligning the network topology with the NVLink domains, we minimize the number of switch hops for the most common collective patterns.
- The Goal: Minimize “east-west” traffic across the spine for the majority of the training step.
- The Reality: This requires precise cabling. In a 32,768 GPU cluster, you are managing thousands of 800G links that must be mapped to specific physical GPU ranks to maintain rail alignment.
Tuning the Collective: NCCL on RoCE
The NVIDIA Collective Communications Library (NCCL) is the secret sauce. For Blackwell, NCCL must be aware of the multi-tier hierarchy. When you run a ncclAllReduce, the library doesn’t just broadcast data. It breaks the operation into:
- Reduce: Aggregating data within the NVLink rack (Tier 1).
- Network Exchange: Moving the reduced data across the RoCE fabric to other racks (Tier 2).
- Broadcast: Distributing the final result back via NVLink.
Optimization: PXN (Proxy-based NVLink)
In Blackwell clusters, we utilize PXN. If GPU 0 needs to send data to a remote rack, but the path through its local NIC is congested, it can “hop” over NVLink to GPU 1 and use GPU 1’s NIC to exit the rack. This turns the entire 72-GPU NVLink domain into a massive, shared pool of network bandwidth.
# Example NCCL environment variables for RoCE Optimization
export NCCL_IB_GID_INDEX=3
export NCCL_IB_TC=128
export NCCL_IB_HCA=^mlx5_bond_0
export NCCL_IB_RETRY_CNT=7
export NCCL_NET_GDR_LEVEL=5 # Enable GPUDirect RDMA across multi-tierSolving the Congestion Crisis: DCQCN and Beyond
The biggest enemy of RoCE in Blackwell clusters is Incast Congestion. Imagine 71 GPUs all trying to send data to a single GPU simultaneously during an All-to-All operation. The switch buffers fill up instantly.
In traditional Ethernet, the switch would just drop the packets. In RoCE, we use Priority Flow Control (PFC) to send a “Pause” frame, telling the sender to stop. But if not tuned correctly, this leads to PFC Storms and Head-of-Line Blocking, where the entire network grinds to a halt.
The Solution: DCQCN (Data Center Quantized Congestion Notification)
DCQCN is the “brain” that manages RoCE congestion. It uses three components:
- Reaction Point (RP): The source NIC that throttles the sending rate.
- Congestion Point (CP): The switch that detects buffer build-up and marks packets (using ECN - Explicit Congestion Notification).
- Notification Point (NP): The destination NIC that sees the ECN mark and sends a “slow down” message back to the source.
Engineering Insight: On Blackwell clusters, we don’t just “turn on” DCQCN. We have to tune the alpha (rate reduction factor) and g (the gain for rate recovery) based on the diameter of the fabric. If alpha is too aggressive, your bandwidth utilization drops. If it’s too passive, you get PFC pauses that spike your tail latency.
The Next Frontier: Packet Spraying and Adaptive Routing
Standard Ethernet uses ECMP (Equal-Cost Multi-Path) to distribute traffic. ECMP hashes the flow (Source IP, Dest IP, Ports) to a specific path. In AI, we have a few “elephant flows” rather than millions of small “mice flows.” If two elephant flows hash to the same path, the link saturates while others sit idle.
To fix this for Blackwell, we are moving toward Packet Spraying (pioneered by technologies like NVIDIA’s Adaptive Routing and the UEC transport).
Instead of sending a whole flow down one path, the hardware breaks the RDMA message into small packets and “sprays” them across all available paths to the destination. They arrive out-of-order and are reassembled by the SuperNIC hardware.
Why this is huge for Blackwell: It allows us to run the RoCE network at 95%+ utilization without creating hotspots. In a 100,000 GPU cluster, this translates to a 20-30% reduction in total training time—saving millions of dollars in compute costs.
Hardware-Offloaded Collectives (SHARPv3)
Finally, we have to talk about In-Network Computing. Why move data to a GPU just to sum it up and move it again?
With SHARPv3 (Scalable Hierarchical Aggregation and Reduction Protocol), the RoCE switches themselves have logic units. As packets from Blackwell GPUs move through the switch, the switch performs the mathematical reduction (the “SUM” in All-Reduce) in-flight.
By the time the data reaches the destination, the computation is already done. This offloads the Blackwell Tensor Cores from doing mundane reductions, allowing them to focus on the heavy lifting of matrix multiplication.
Putting it All Together: The Optimized Stack
Optimizing RoCE for Blackwell isn’t about a single “magic” setting. It’s a full-stack engineering effort:
- Physical Layer: Rail-optimized cabling to align NVLink domains with RoCE rails.
- Link Layer: PFC and ECN tuning to prevent “Pause” frames while managing buffers.
- Transport Layer: DCQCN parameters tuned for the specific RTT (Round Trip Time) of the data center scale.
- Collective Layer: NCCL tuned with PXN and Sharpv3 to leverage the hierarchy of the GB200 rack.
The Blackwell era is moving us toward a world where the data center is the computer. The network is no longer just a pipe; it is a backplane. As we push toward 100K GPU clusters, the engineers who master the intricacies of RoCE and multi-tier collectives will be the ones who define the future of AI.
The hype for Blackwell is real, but remember: the silicon might provide the TFLOPS, but the network provides the scale.