It’s 3:00 AM. You’re staring at a Grafana dashboard monitoring a training run for a trillion-parameter Large Language Model (LLM). Everything looks perfect—until it doesn't. Suddenly, the MFU (Model Flops Utilization) drops from 60% to 5%. The loss curve flatlines. Tens of millions of dollars of compute are spinning their wheels, burning megawatts of power, while thousands of GPUs wait for a single missing packet.
Welcome to the world of 100k+ GPU clusters.
At this scale, the network is no longer just "the plumbing." It is the computer. When you are orchestrating 100,000 H100s or B200s, the traditional rules of data center networking don't just bend—they shatter. For years, the high-performance computing (HPC) world belonged to InfiniBand. But as AI scales to the stratosphere, RoCE v2 (RDMA over Converged Ethernet) has emerged as the challenger.
The promise? Massive scale and lower costs using the Ethernet ecosystem we’ve built over 40 years. The catch? Ethernet was never designed for this level of synchronicity. To make RoCE v2 work at 100k-GPU scale, we have to solve the "Congestion Control" and "Tail Latency" monsters that haunt the dreams of every site reliability engineer in the AI space.
Let’s dive into the guts of how we build and optimize these massive AI fabrics.
The Infrastructure: Why 100k GPUs Changes Everything
In a standard enterprise data center, a "large" network might handle a few thousand servers. AI training is different. We are talking about non-blocking, high-bandwidth, low-latency fabrics where every GPU must talk to every other GPU with microsecond-level precision.
The Rail-Optimized Topology
If you build a 100,000 GPU cluster using a standard Clos (leaf-spine) architecture, your cabling becomes a nightmare, and your latency gets unpredictable. The industry has shifted toward Rail-Optimized designs.
In a rail-optimized setup, if a server has 8 GPUs, each GPU connects to a different "rail" (a separate network fabric).
- GPU 0 across all 12,500 nodes connects to Fabric 0.
- GPU 1 connects to Fabric 1.
- And so on.
When you perform an All-Reduce operation—the bread and butter of distributed training—the GPUs on the same "rail" communicate with each other. This localizes traffic and minimizes the number of switch hops, but it puts an astronomical burden on the network to be perfectly balanced. If "Rail 3" experiences congestion, the entire 100,000 GPU cluster slows down to the speed of that congested rail.
The RoCE v2 Architecture: RDMA’s Ethernet Skin
RDMA (Remote Direct Memory Access) is the magic that allows one GPU to write directly into the memory of another GPU without involving the CPU, the OS kernel, or unnecessary memory copies.
RoCE v2 wraps these RDMA verbs inside UDP/IP packets. This allows RDMA to travel over standard Ethernet switches.
The Problem: Ethernet is Lossy
Standard Ethernet is a "best-effort" medium. If a switch buffer gets full, it just drops the packet. In the world of TCP, this is fine—the protocol will eventually retransmit. But in AI training, a single packet drop can trigger a catastrophic retransmission timeout that stalls the entire global training step.
To prevent this, RoCE v2 relies on PFC (Priority Flow Control).
Solving the Congestion Beast: From PFC to DCQCN
PFC is a hop-by-hop flow control mechanism. When a switch port's buffer starts to fill up, it sends a "PAUSE" frame to the sender, telling it to stop for a few microseconds.
At 100k scale, PFC is a double-edged sword. It prevents packet loss, but it creates two new nightmares:
- Head-of-Line (HoL) Blocking: One slow flow causes the switch to pause an entire port, punishing perfectly healthy flows that happened to be in the same queue.
- PFC Deadlocks (and Storms): In complex topologies, a chain of PAUSE frames can loop back on itself, freezing the entire network in a deadlock.
Enter DCQCN (Data Center Quantized Congestion Control)
To scale beyond a few thousand nodes, we can't rely on PFC alone. We need an end-to-end congestion control algorithm. DCQCN is the current industry standard, combining ECN (Explicit Congestion Notification) and PFC.
Here is how the DCQCN handshake works under the hood:
- The Switch (The Observer): When a switch buffer exceeds a certain threshold (K-min), it marks the IP header of the packets with a "Congestion Experienced" (CE) bit.
- The Receiver (The Messenger): The destination NIC sees the CE bit and sends a CNP (Congestion Notification Packet) back to the source NIC.
- The Source (The Governor): The source NIC receives the CNP and immediately throttles its transmission rate. It then uses a timer-based and byte-counter-based algorithm to slowly ramp the speed back up once the CNPs stop arriving.
The Code Perspective: Tuning the DCQCN Parameters
Optimizing DCQCN for 100k GPUs isn't just about turning it on; it’s about the math of the "Alpha" (reduction factor) and "Beta" (increase factor).
# Example: Setting DCQCN parameters on a Mellanox/NVIDIA ConnectX-7 NIC
# Using mlxconfig to tune the rate reduction and increase
# We want a fast reduction but a cautious, linear increase to avoid oscillations.
# Set the target rate reduction alpha
mlxconfig -d /dev/mst/mt4129_pciconf0 set ROCE_CC_ALPHAI_RT_7=10
# Set the rate increase timer (high values = slower recovery)
mlxconfig -d /dev/mst/mt4129_pciconf0 set ROCE_CC_RATE_REDUCE_MONITOR_PERIOD_7=4
# Set ECN marking threshold on the switch (Arista/Mellanox style)
# If buffer > 150KB, start marking ECN
switch(config)# random-detect ecn minimum-threshold 150 kbytes maximum-threshold 1500 kbytes
In a 100k GPU cluster, the "Target Rate" recovery must be incredibly precise. If you recover too fast, you trigger another PFC PAUSE. If you recover too slowly, you leave precious bandwidth on the table.
The "Tail Latency" Killer: Micro-bursts and Incast
In AI training, we care about the 99.99th percentile (P99.99) latency. Why? Because of the Bulk Synchronous Parallel (BSP) nature of training.
In a training step, all 100,000 GPUs do their math, and then they all exchange gradients. This is an Incast event. Thousands of GPUs send data to a single node simultaneously. If one packet gets stuck behind a micro-burst of traffic and takes 1ms instead of 10μs, the entire 100,000 GPU cluster waits for that 1ms.
One slow packet = 100,000 idle GPUs.
Solving Incast with Packet Spraying
Standard Ethernet uses ECMP (Equal-Cost Multi-Path) to distribute traffic. ECMP hashes the 5-tuple (Source IP, Dest IP, Port, etc.) to choose a path. The problem? AI flows are "elephants"—massive, long-lived streams of data. If two elephant flows hash to the same physical link, that link gets crushed while others sit idle.
The solution being deployed at the 100k scale (like in NVIDIA Spectrum-X or Broadcom Jericho3-AI) is Adaptive Routing and Packet Spraying.
Instead of sending an entire flow down one path, the hardware breaks the RDMA message into small packets and "sprays" them across every available path in the fabric.
- Hardware Reordering: The receiving NIC is responsible for putting the packets back in order.
- Result: 95%+ link utilization and the elimination of "hot spots" in the network.
The Hype vs. Reality: Ethernet’s "InfiniBand Moment"
There has been massive hype around the "Ultra Ethernet Consortium" (UEC) and the idea that Ethernet is "killing" InfiniBand. Meta’s massive 24k and now 100k+ GPU clusters are the primary proof points for RoCE v2.
Why the shift?
- Supply Chain: There’s only one major vendor for InfiniBand. There are dozens for Ethernet.
- Economics: 800G Ethernet is reaching commodity status faster than proprietary interconnects.
- Flexibility: Ethernet allows you to use standard monitoring tools (SNMP, gRPC, streaming telemetry) that the networking world has spent decades perfecting.
The Reality: RoCE v2 is harder to manage than InfiniBand. InfiniBand is "lossless by design" at the credit-based link layer. Ethernet is "lossy by design" and requires you to layer complex congestion control algorithms on top to simulate losslessness. If your DCQCN parameters are off by 5%, your 100k GPU cluster might perform 20% worse.
The Engineering Frontier: Programmable Congestion Control (PCC)
We are moving away from fixed algorithms like DCQCN and toward PCC (Programmable Congestion Control).
Companies like Google (with Swift) and Microsoft are experimenting with using RTT (Round Trip Time) as the primary signal for congestion, rather than switch buffer marks (ECN).
Why RTT?
ECN is binary (marked or not marked). RTT is granular. By measuring the nanosecond-level changes in how long a packet takes to return an ACK, the NIC can sense a buffer building up before the switch even thinks about marking an ECN bit.
PCC Logic in a Nutshell:
# Conceptual PCC logic running on the NIC firmware
def update_rate(current_rate, measured_rtt, target_rtt):
if measured_rtt > target_rtt:
# We are seeing queue buildup, throttle back
new_rate = current_rate * (1 - (measured_rtt - target_rtt) / measured_rtt)
else:
# Path is clear, ramp up slowly
new_rate = current_rate + additive_increase_constant
return new_rate
At 100k scale, these algorithms are often implemented in P4 or specialized hardware state machines to ensure they can make rate-limiting decisions every few nanoseconds.
Deep-Scale Monitoring: The "Invisible" Drops
In a 100k GPU fabric, you cannot rely on "polling" counters every 30 seconds. You need In-band Network Telemetry (INT).
INT inserts a small metadata header into the actual data packets. As a packet traverses the 100,000 GPU fabric, each switch adds its:
- Switch ID
- Ingress/Egress Timestamp
- Queue Depth
- Link Utilization
When the packet reaches the destination, the NIC strips this metadata and sends it to a massive telemetry collector. This allows you to reconstruct the exact path of a "slow" packet and identify which specific switch buffer in a sea of 5,000 switches caused the delay.
The RoCE v2 Checklist for 100k GPUs
If you’re building at this scale, these are the "non-negotiables":
- PFC + ECN (DCQCN): Mandatory. Do not attempt RoCE v2 at scale without end-to-end congestion control.
- Hardware-Based Packet Spraying: Forget ECMP. You need a fabric capable of adaptive routing to handle elephant flows.
- Buffer Tuning: Large-scale AI training requires deep buffers on leaf switches to handle incast, but shallow, ultra-fast buffers on the backbone to keep latency low.
- Zero-Touch RoCE (ZTR): Implementing ZTR helps reduce the configuration complexity of PFC by making the protocol more resilient to packet loss, though it's still an emerging standard.
- Rail-Optimization: Align your network topology with your GPU topology. If your software thinks GPUs are neighbors, they should be neighbors on the wire.
Summary: The Network is the Computer
Optimizing RoCE v2 for a 100,000 GPU cluster is perhaps the most difficult challenge in modern systems engineering. It requires a deep understanding of physics (signal integrity over DAC cables), mathematics (congestion control feedback loops), and distributed systems (collective communication primitives).
The era of "set it and forget it" networking is over. To train the next generation of AI, we aren't just moving packets; we are orchestrating a massive, 100,000-part symphony where a single late note can ruin the entire performance.
By mastering PFC deadlocks, DCQCN tuning, and packet spraying, we move closer to a future where Ethernet isn't just a "good enough" alternative to InfiniBand—it’s the foundation for the largest machines ever built by humanity.