Imagine you are orchestrating a symphony with 50,000 musicians. Now, imagine that for the symphony to sound coherent, every single musician must be able to whisper a secret to any other musician in the room, nearly instantaneously, without interrupting the melody.
In the world of Generative AI, this isn’t a metaphor. It’s the daily reality of training Large Language Models (LLMs). When we talk about training a model like GPT-4 or Llama 3, we aren’t just talking about raw compute; we are talking about a massive, high-speed communication problem. For a decade, the Fat Tree (Clos) topology has been the undisputed king of the data center. It was reliable, predictable, and scalable.
But as we push toward the “Million-GPU Cluster,” the Fat Tree is hitting a physical and economic wall. The sheer volume of optics, the complexity of the cabling, and the “stranded capacity” of traditional networking are forcing us to rethink the very geometry of the data center.
Welcome to the era of disaggregated, rail-optimized, and optically-switched topologies. In this deep dive, we’re going beyond the three-tier Clos to explore the bleeding edge of AI networking.
The “Fat Tree” Tax: Why We’re Looking for an Exit
To understand where we’re going, we have to acknowledge where we are. The Fat Tree (specifically the 3-stage Clos) has been the gold standard because it provides non-blocking, any-to-any connectivity. If you have 2,000 servers, any server can talk to any other server at full line rate.
However, AI workloads are not “any-to-any” in a random sense. They are highly structured.
In an AI training run, GPUs perform a cycle:
- Compute (Forward/Backward pass)
- Communicate (All-Reduce or All-to-All to sync gradients)
- Repeat
During the “Communicate” phase, the network is slammed with “Elephant Flows”—massive, long-lived bursts of data. In a traditional Fat Tree, these flows often collide. Even with Equal-Cost Multi-Path (ECMP) routing, two massive flows might get hashed to the same uplink, causing congestion while other links sit idle.
Furthermore, the radix (port count) of switches is struggling to keep up. To build a non-blocking Fat Tree for 32,000 GPUs at 800Gbps, you need a staggering amount of optical transceivers. At current prices, the networking fabric can account for nearly 20-30% of the total cluster cost.
We are paying a “Fat Tree Tax”—buying expensive bandwidth and hardware that isn’t always optimized for the specific way AI models learn.
1. Rail-Optimized Design: Aligning Silicon with Geometry
The first major shift in next-gen AI networking isn’t a new topology, but a new way of mapping GPUs to the network. This is known as Rail-Optimization.
In a standard dense compute node (like an NVIDIA DGX H100), you have 8 GPUs. In a traditional setup, you might just plug these into the nearest Leaf switch. But in a rail-optimized design, we treat each “rail” (the set of GPUs in the same position across all nodes) as a separate network slice.
How it Works:
- GPU 0 from every server in a row connects to Leaf Switch A.
- GPU 1 from every server connects to Leaf Switch B.
- …and so on, up to GPU 7.
Why this is a Game Changer:
Most AI collective operations (like All-Reduce) happen among the “rank-equivalent” GPUs across nodes. By ensuring that all “GPU 0s” are on the same leaf or the same minimal-hop path, we minimize the number of hops a packet takes during the most critical phases of training.
The Technical Substance: This reduces the load on the Spine layer. If your collective communication can be contained within the “Rail,” you don’t need to traverse the entire Fat Tree to sync gradients. This effectively creates a “disaggregated” network where the heavy lifting is done at the edge, reducing latency and power consumption.
2. The Return of the Torus (and the Rise of the Dragonfly)
While InfiniBand-based Fat Trees dominate the headlines, some of the world’s most powerful AI supercomputers—specifically Google’s TPU pods—use an entirely different geometry: the 3D (or Higher-D) Torus.
The Torus Architecture
In a Torus topology, each node is connected to its immediate neighbors in a grid (X, Y, and Z dimensions).
- Pros: No expensive central Spine switches. It’s incredibly cheap to scale because you only buy cables to connect neighbors.
- Cons: The “diameter” of the network (the maximum distance between any two nodes) can be high. If Node A needs to talk to Node Z on the other side of the data center, it has to hop through many intermediate nodes.
Enter the Dragonfly
To solve the diameter problem of the Torus while keeping the cost benefits, engineers are turning to Dragonfly topologies.
A Dragonfly topology groups routers into “cliques.” Inside the clique, everything is all-to-all. But then, each clique has a few direct “long-haul” connections to every other clique in the system.
Clique A [Nodes 1-4] --(Long-haul)--> Clique B [Nodes 5-8]
| |
+------------(Long-haul)--------------+Why it matters for AI: Dragonfly minimizes the number of expensive optical cables. In a Fat Tree, roughly 50% of your cables are “long” (going from Leaf to Spine). In a Dragonfly, that number drops significantly. The challenge? Routing. Because there are multiple ways to get from Clique A to Clique B (direct or via Clique C), the network needs Adaptive Routing to steer traffic around congestion in real-time.
3. Optical Circuit Switching (OCS): Moving Mirrors, Not Packets
Perhaps the most radical departure from traditional networking is Google’s use of Apollo, an Optical Circuit Switching (OCS) fabric.
In a traditional network, every switch is “Packet Switched.” The switch looks at every individual packet, reads the header, and decides where to send it. This requires converting light (from the fiber) into electricity, processing it in silicon, and converting it back to light. This process consumes massive amounts of power and adds latency.
OCS does away with the conversion.
The Tech: MEMS and Mirrors
An OCS switch (like the ones used in TPU v4 and v5p) uses an array of tiny, steerable MEMS (Micro-Electro-Mechanical Systems) mirrors.
- An optical fiber enters the switch.
- The light hits a tiny mirror.
- A software controller tilts that mirror to reflect the light directly into a specific output fiber.
The Engineering Win:
- Zero Packet Loss: There are no buffers to overflow.
- Protocol Agnostic: It doesn’t care if you’re running Ethernet, InfiniBand, or a custom protocol.
- Dynamic Reconfiguration: If a rack of GPUs fails, the software can literally “re-wire” the data center in milliseconds by tilting mirrors, creating a new topology that bypasses the dead nodes.
The Catch: You can’t change the mirror position for every packet (the mirrors are “slow”—taking milliseconds to move). This means OCS is used for the topology layer, not the packet-routing layer. It’s “disaggregated” networking where the physical layer itself is programmable.
4. Disaggregated Fabrics and CXL: The Rack is the New Server
When we talk about “Disaggregated Topologies,” we often mean moving beyond the idea of a “server” as a fixed box. With the advent of CXL (Compute Express Link), we are looking at a future where memory and compute are decoupled.
In current architectures, if a GPU runs out of VRAM, the training job crashes (the dreaded OOM - Out of Memory). In a disaggregated topology using a CXL Fabric, a GPU can reach across the network to a “Memory Pool” and borrow capacity.
The Technical Challenge: The “Speed of Light” Problem
CXL over a fabric requires sub-microsecond latency. This is why we are seeing a shift toward “Rack-Scale” design. Instead of thinking about thousands of individual servers, companies are designing the entire rack as a single “Super-Node.”
- Internal Fabric: All-to-all copper backplane (ultra-low latency).
- External Fabric: High-radix optical uplinks.
By using a Non-Transparent Bridge (NTB) or CXL switches, we can create a shared memory space across 64 or 128 GPUs. This changes the network topology from a “communication pipe” to a “memory bus.”
5. Solving the “Elephant Flow” Problem: Packet Spraying and RoCE v2
Regardless of the physical topology, the way we move data over the wires is changing. The industry is currently split between InfiniBand (the incumbent) and RoCE v2 (RDMA over Converged Ethernet).
The biggest problem in AI Ethernet is Hash Polarization. In standard ECMP, a “flow” (a stream of packets between two IPs) is always sent over the same path to prevent out-of-order packets. In AI, one “flow” might be 100GB. If two 100GB flows hit the same link, you get a bottleneck.
The Solution: Packet Spraying (Direct Data Placement)
Next-gen topologies are moving toward Adaptive Routing and Packet Spraying. Instead of sending a flow down one path, the NIC (Network Interface Card) breaks the flow into small chunks and “sprays” them across every available path in the network—even if they arrive out of order.
# Conceptual pseudocode for a Packet-Spraying Logic in a SmartNIC
def send_payload(payload, paths):
chunks = split_into_mtu_sized_chunks(payload)
for i, chunk in enumerate(chunks):
# Dynamically select the path with the lowest queue depth
best_path = min(paths, key=lambda p: p.current_latency)
send_to_fabric(chunk, path=best_path, sequence_id=i)
# On the receiving end, the hardware re-assembles based on sequence_idThis requires specialized hardware (like NVIDIA’s Spectrum-X or Broadcom’s Jericho3-AI) that can handle reassembly at 800Gbps. This effectively turns the network into a fluid fabric where no single link is ever the bottleneck.
Scaling to the “Frontier”: The Engineering Curiosities
As we look toward 100,000+ GPU clusters, a few weird engineering challenges are popping up that weren’t an issue five years ago:
The “Speed of Light” is Too Slow
At 800Gbps, the “bits” are physically very short in the fiber. If your fiber optic cable is 100 meters long, the time it takes for light to travel that distance is ~500 nanoseconds. In AI training, where every microsecond of “Tail Latency” (the 99th percentile slowest packets) can stall the entire cluster, we are seeing a push for shorter cables. This is leading to “dense packing,” where racks are literally curved or circular to minimize cable length.
Reliability and the “Blast Radius”
In a 3-tier Fat Tree, the failure of a Spine switch can take down 10% of your cluster’s bandwidth. In disaggregated topologies like Dragonfly, the Blast Radius is different. A single link failure might only slightly increase the “path stretch” for certain nodes. We are moving toward “Graceful Degradation”—the network keeps running, just slightly slower, rather than the “all-or-nothing” nature of InfiniBand subnets.
The Power Wall
We are approaching a point where the optics consume more power than the switching silicon. This is driving research into CPO (Co-Packaged Optics), where the laser and the fiber connection are moved directly onto the same package as the Ethernet chip. This eliminates the need for power-hungry “Retimers” that boost the signal across the PCB.
The Landscape Ahead
We are witnessing a “Cambrian Explosion” of network diversity. For the first time in two decades, there isn’t just one way to build a data center.
- Hyperscalers (Google/Meta) are building custom, optically-switched, or rail-optimized fabrics to squeeze every bit of efficiency out of their proprietary silicon.
- Enterprise AI/Cloud Providers are pushing the limits of Ethernet with packet spraying and RoCE v2 to maintain flexibility.
- Research Labs are experimenting with Torus and Dragonfly designs to minimize the “Optics Tax.”
The Fat Tree served us well during the era of the “General Purpose Cloud.” But the era of the “AI Supercomputer” demands something more organic, more reconfigurable, and more tightly coupled with the silicon it serves.
The future of the data center isn’t a tree; it’s a high-dimensional, adaptive, and potentially light-steered fabric. As we scale toward models with trillions of parameters, the network is no longer just “the plumbing”—it is the backbone of the intelligence itself. If the GPUs are the neurons, the next generation of disaggregated topologies is the high-speed synaptic network that makes the “brain” possible.