There is a quiet, frantic revolution happening inside the windowless monolithic structures that dot the landscapes of Northern Virginia, Dublin, and Singapore.
If you were to step inside a hyperscale data center ten years ago, the “network” was essentially a utility—the plumbing that carried HTTP requests from a load balancer to a web server. But today, the network has undergone a fundamental metamorphosis. In the era of Large Language Models (LLMs) and massive-scale distributed computing, the network is no longer just “the plumbing.” The network is the backplane of the world’s largest distributed computer.
When you are training a model with 1.8 trillion parameters across 20,000 H100 GPUs, the bottleneck isn’t just the TFLOPS on the chip; it’s the speed at which those chips can talk to each other. We are no longer talking about “gigabits per second.” We are firmly in the era of Terabit-scale interconnects, where a single switch ASIC handles 51.2 Tbps of throughput, and the margin for error in congestion management is effectively zero.
In this deep dive, we’re going to peel back the layers of the modern hyperscale fabric. We’ll explore the move from NRZ to PAM4 signaling, the physics of 800G/1.6T optics, the “lossless” Ethernet debate, and the sophisticated congestion control algorithms that keep the global AI engine from grinding to a halt.
The Death of the Three-Tier Tree and the Rise of the High-Radix Clos
To understand where we are, we have to look at the wreckage of where we were. Traditional data center networks were built on a hierarchical tree model: Access, Aggregation, and Core. This worked perfectly for “North-South” traffic (users talking to servers).
However, as applications became microservices-oriented and AI training became the primary workload, traffic patterns flipped. Now, 90% of traffic is “East-West” (servers talking to servers). In a hierarchical tree, East-West traffic has to travel “up” the tree and back “down,” creating massive bottlenecks at the core.
The Clos (Spine-Leaf) Revolution
Hyperscalers moved to the Clos architecture, commonly known as Spine-Leaf. In this topology, every Leaf switch connects to every Spine switch. This provides:
- Fixed Latency: Every server is exactly three hops away from any other server.
- Linear Scalability: Need more bandwidth? Add another Spine. Need more ports? Add another Leaf.
- Resiliency: If one Spine fails, the fabric only loses a fraction of its total capacity.
But as we hit the Terabit era, the challenge isn’t just the topology—it’s the Radix (port density). To build a cluster for 50,000 GPUs, you need switches with massive radix to keep the “hop count” low. If you have to go through five layers of switches instead of three, your tail latency (p99) skyrockets, and your AI training job takes three weeks longer.
Today’s gold standard is the Broadcom Tomahawk 5 or the Nvidia Spectrum-4, switch ASICs capable of 51.2 Tbps in a single RU. This allows for 64 ports of 800GbE or 128 ports of 400GbE. We are already seeing the blueprints for 102.4 Tbps chips that will usher in the 1.6 Terabit port era.
The Physical Wall: From NRZ to PAM4 and the 224G SerDes
Engineering at this scale is a fight against the laws of physics. For decades, we used NRZ (Non-Return-to-Zero) signaling—essentially “on” or “off” (1 or 0). But as we tried to push NRZ to 50Gbps and beyond, the signal degradation became unmanageable. The “eye diagram” (the visual representation of signal integrity) simply closed.
To solve this, the industry moved to PAM4 (4-level Pulse Amplitude Modulation). Instead of two levels, PAM4 uses four signal levels, allowing each clock cycle to carry two bits of data instead of one.
| Feature | NRZ | PAM4 |
|---|---|---|
| Bits per Symbol | 1 bit | 2 bits |
| Signal Levels | 0, 1 | 00, 01, 10, 11 |
| Bandwidth | 1x | 2x |
| Complexity | Low | High (Requires FEC) |
The SerDes Challenge
The heart of this transition is the SerDes (Serializer/Deserializer). This is the hardware block that converts parallel data from the chip into serial data for the copper or fiber. We are currently transitioning from 112G SerDes to 224G SerDes.
At 224Gbps per lane, the copper traces on a PCB act like antennas. The signal disappears into the board (attenuation) or leaks into adjacent lanes (crosstalk). This is why you see hyperscalers moving toward Co-Packaged Optics (CPO). Instead of plugging a transceiver into the front of a switch, we are moving the optical engines directly onto the chip substrate. This reduces the distance the electrical signal has to travel from inches to millimeters, drastically cutting power consumption—which is critical when a single switch can pull 500+ watts.
The “Lossless” Dilemma: RoCEv2 vs. InfiniBand
In a standard web environment, Ethernet is “lossy.” If a packet is dropped due to congestion, TCP notices and retransmits it. For a Netflix stream, a few milliseconds of retransmission is invisible.
For AI training, a dropped packet is a catastrophe.
AI workloads use collective communication primitives like All-Reduce. In an All-Reduce operation, thousands of GPUs must synchronize their gradients. If one packet is dropped, the entire 20,000-GPU cluster stalls, waiting for that one packet to be retransmitted. This is the “Incast” problem, and it can drop effective link utilization from 95% to 10%.
InfiniBand: The Incumbent
InfiniBand has long been the king of the supercomputer. It is a credit-based, lossless architecture from the ground up. A sender won’t send data unless the receiver confirms it has the buffer space to hold it. No drops, ultra-low latency.
RoCEv2: The Challenger
Hyperscalers (Meta, Google, Microsoft) generally prefer Ethernet because of the massive ecosystem and lower cost. To make Ethernet work for AI, they use RoCEv2 (RDMA over Converged Ethernet).
RoCEv2 allows a GPU to write directly into the memory of another GPU across the network, bypassing the CPU and the OS kernel (Remote Direct Memory Access). But to make this work, we have to “force” Ethernet to be lossless using two key mechanisms:
- PFC (Priority Flow Control): When a switch buffer fills up, it sends a “PAUSE” frame to the sender.
- ECN (Explicit Congestion Notification): The switch marks packets when it’s getting crowded, telling the end hosts to slow down before drops happen.
Micro-bursts and the Ghost in the Fabric: Congestion Management
Even with RoCEv2, the network faces a silent killer: Micro-bursts. These are surges of traffic that last only microseconds—too fast for traditional monitoring tools to see—but long enough to overflow a switch buffer and trigger a PFC PAUSE frame.
If not managed perfectly, you get a PFC Storm. A switch pauses its neighbor, which pauses its neighbor, and suddenly your entire data center fabric is deadlocked in a “pause-spread” chain reaction.
Modern Congestion Control Algorithms
This is where the software engineering becomes incredibly sophisticated. We’ve moved beyond simple Reno/Cubic TCP.
- DCQCN (Data Center Quantized Congestion Notification): This is the standard for RoCEv2. It combines ECN and PFC with a complex state machine on the NIC to calculate the exact rate at which it should inject traffic into the fabric.
- HPCC (High Precision Congestion Control): Developed by Alibaba, this uses INT (In-band Network Telemetry). Instead of guessing based on dropped packets, the switches stamp every packet with their exact queue depth and link utilization. The receiver sees this and tells the sender exactly how many Gbps it can send.
The Shift to Adaptive Routing
Traditionally, Ethernet uses ECMP (Equal-Cost Multi-Path) to spread traffic. ECMP hashes packet headers (Source IP, Dest IP, etc.) to choose a path. If two huge “elephant flows” hash to the same path, that link gets crushed while others sit idle.
The cutting edge is Adaptive Routing. Modern switch ASICs (like the Tomahawk 5 or Nvidia’s Spectrum-4) can monitor the load on their exit ports in real-time. If a packet was supposed to go out on Port 1, but Port 1 is congested, the switch hardware will reroute it to Port 2 on the fly. This happens in nanoseconds, at the hardware level, without the CPU ever knowing.
Programmability and Visibility: The P4 Revolution
You cannot fix what you cannot see. In the old days, network visibility meant SNMP polls every 5 minutes. In a Terabit fabric, 5 minutes of data is an eternity.
The rise of the P4 programming language and programmable pipelines has changed the game. We can now treat the switch pipeline like code. If we want to track the latency of every single packet in a flow, we can write a P4 program to do it.
// A simplified snippet of what In-band Network Telemetry looks like in P4
control egress(inout headers hdr, inout metadata meta) {
apply {
if (hdr.int_header.isValid()) {
hdr.int_data.push_front({
switch_id: MY_ID,
ingress_port: meta.ingress_port,
egress_port: meta.egress_port,
queue_depth: standard_metadata.deq_qdepth,
timestamp: standard_metadata.egress_global_timestamp
});
}
}
}By embedding metadata into the packet itself as it traverses the fabric, engineers can reconstruct a “holographic” view of the network’s state at any microsecond. This allows for Automated Remediation: if a specific optical lane is showing an increased bit-error rate (BER), the control plane can automatically drain traffic from that link before it fails and causes a training stall.
The AI Hype vs. The Infrastructure Reality
The tech world is currently obsessed with “The Model.” Everyone talks about GPT-5, Gemini, or Llama. But for those of us in the trenches, the “Model” is just the payload.
The hype cycle often ignores the “Network Tax.” As models grow, the percentage of time spent on communication (waiting for the network) versus computation (math on the GPU) increases. If your network isn’t designed for Terabit-scale, your $500 million GPU cluster might spend 40% of its time just sitting idle, waiting for packets.
This is why we are seeing a massive surge in DCN (Data Center Network) specialization. We are moving away from “General Purpose” networks. Hyperscalers are now building two separate networks:
- The Front-end Fabric: Standard Ethernet for storage, management, and user traffic.
- The Backend (Compute) Fabric: A hyper-optimized, high-radix, low-latency, lossless fabric (InfiniBand or Ultra-Ethernet) dedicated solely to GPU-to-GPU communication.
The Road to 1.6T and Beyond
Where do we go from here? The roadmap for the next three years is already written in the silicon.
- 1.6 Terabit Ethernet: We will see 1.6T ports becoming the backbone of the next generation of AI clusters (using 16 lanes of 100G or 8 lanes of 200G).
- The Ultra Ethernet Consortium (UEC): A massive industry push (AMD, Arista, Broadcom, Google, Meta) to evolve Ethernet into a protocol that can finally kill InfiniBand by adding native support for out-of-order delivery and more flexible congestion control.
- Optical Switching: Eventually, even the electrical switch ASIC will become a bottleneck. We are seeing research into All-Optical Switching (OCS), where MEMS mirrors physically move to steer beams of light, bypassing the need for electrical conversion entirely (Google is already doing this with their Apollo fabric).
Final Thoughts from the Engineering Frontline
The evolution of data center fabrics is a testament to the “Scaling Laws.” As long as we find that more data and more compute lead to more intelligent models, the pressure on the network will be relentless.
We have moved from a world of “best-effort” packet delivery to a world of nanosecond-precision orchestration. Building for Terabit-scale isn’t just about faster chips; it’s about a holistic rethink of the entire stack—from the way electrons move through a 224G SerDes to the way a P4 program monitors a queue, to the way a global congestion algorithm balances a trillion-parameter flow.
The next time you ask an AI a question and get a response in seconds, remember: there is a Terabit fabric behind that curtain, humming at the speed of light, fighting a constant war against congestion to make it all possible.
Are you ready for the 1.6T transition? Because the silicon is already on the way.