In the early 2010s, a “large” distributed system meant a few dozen nodes syncing over Gigabit Ethernet. Today, we are building cathedrals of compute. To train a trillion-parameter Large Language Model (LLM), we aren’t just plugging servers into a rack; we are building a singular, planetary-scale machine.
When you look at the floor plan of a modern AI supercluster—be it Meta’s Grand Teton or Microsoft’s Eagle—the most striking feature isn’t the GPUs themselves. It’s the cabling. Thick bundles of InfiniBand, the complex mesh of NVLink, and the emerging promise of CXL are the veins and arteries of the beast. Without them, the H100s and B200s are just very expensive paperweights.
If you want to understand why AI scaling hasn’t hit a wall yet, you have to look at the interconnect. This is a deep dive into the high-bandwidth trinity that makes the “World Model” possible.
The Bottleneck: Why “Good Enough” Networking Died
In traditional web-scale engineering, we optimize for North-South traffic (client to server). In AI training, we are 100% focused on East-West traffic (GPU to GPU).
When training an LLM using Data Parallelism or Tensor Parallelism, every GPU must constantly share its weight gradients with every other GPU. If one GPU is 10 microseconds slower than the others, the entire cluster—thousands of chips—idles, waiting for that one straggler. This is the Tail Latency problem, and in a cluster of 100,000 GPUs, it is the difference between a model taking three months to train or three years.
To solve this, we have moved beyond the standard OSI model. We are now in the era of Memory-Centric Interconnects.
1. NVLink: The Local Nervous System
If you open an NVIDIA DGX H100, you won’t see traditional PCIe lanes connecting the GPUs. You see the NVSwitch fabric.
The Architecture of Scale-Up
NVLink is not a network protocol in the sense that Ethernet is; it is a proprietary, high-speed, point-to-point link designed to provide memory coherence between GPUs.
In the Blackwell (B200) generation, NVLink 5.0 delivers a staggering 1.8 TB/s of bidirectional bandwidth per GPU. To put that in perspective, that’s roughly 18x the bandwidth of a high-end PCIe Gen5 x16 slot.
Why it matters: When we perform Tensor Parallelism, we split a single layer of a neural network across multiple GPUs. This requires such frequent communication that if it went over a traditional network, the latency would destroy performance. NVLink makes 8 or 72 GPUs act as one giant, monolithic processor with a unified memory pool.
The NVSwitch Physical Layer
The magic happens at the NVSwitch level. Instead of a simple “all-to-all” wire mesh, NVSwitch acts as a non-blocking crossbar.
- SHARP (Scalable Hierarchical Aggregation and Reduction Protocol): This is the “secret sauce.” Instead of the GPUs calculating the average of their gradients (the “All-Reduce” operation), the switch itself does the math. By offloading collective communications to the hardware fabric, we reduce the amount of data traversing the wires by 50%.
2. InfiniBand: The Fabric of Reality
Once you move beyond a single rack (the “Scale-Up” limit), NVLink historically struggled with distance. This is where InfiniBand (IB) takes over.
While the “Ethernet vs. InfiniBand” war has been raging for decades, in the world of AI, InfiniBand is currently the undisputed king for one primary reason: RDMA (Remote Direct Memory Access).
Zero-Copy, Zero-CPU
In a standard TCP/IP stack, a packet has to go through the kernel, get buffered, be handled by the CPU, and eventually reach the application. This is “High Jitter.”
InfiniBand allows GPU A in Rack 1 to write directly into the memory of GPU B in Rack 500 without ever bothering the host CPU.
- Protocol: InfiniBand uses a credit-based flow control. It is a “lossless” fabric. Unlike Ethernet, which drops packets when congested and asks for a re-transmit (causing massive latency spikes), InfiniBand throttles at the source.
- Adaptive Routing: Modern IB switches (like the Quantum-2) look at the buffer levels of all available paths in real-time. If one cable is congested, the packet is dynamically rerouted.
The Fat-Tree Topology
To scale to 100,000 GPUs, we use a non-blocking Fat-Tree topology.
- Leaf Switches: Connect directly to the H100 nodes.
- Spine Switches: Interconnect the Leaf switches.
- Core Switches: Interconnect the Spines.
Because IB offers flat, deterministic latency, an AI researcher can treat the entire 10-acre data center as a single computer.
3. CXL: The Great Memory Liberator
While NVLink and InfiniBand handle bandwidth, Compute Express Link (CXL) is here to solve the Memory Wall.
LLMs are hungry. A 1.8-trillion parameter model doesn’t just need FLOPs; it needs massive amounts of High Bandwidth Memory (HBM). But HBM is expensive and physically limited by the size of the GPU die.
CXL 2.0/3.0: Memory Pooling
CXL is built on top of the PCIe Gen5/6 physical layer but introduces a low-latency protocol that allows for Memory Expansion and Memory Pooling.
Imagine you have a cluster where Node A is doing heavy computation and is out of VRAM, while Node B is idling. Historically, Node A would crash with an “Out of Memory” (OOM) error.
- CXL.mem: Allows the CPU and GPU to access external pools of DRAM as if they were local.
- The Use Case: For LLM inference (serving the model), we often need to store the K-V Cache (the “memory” of the current conversation). The K-V cache for long-context windows (like 1M tokens) is massive. CXL allows us to offload this cache to a cheaper, shared pool of DDR5 memory, freeing up the lightning-fast HBM for the actual matrix multiplications.
Why the Hype?
CXL is the first time in history that the industry (Intel, AMD, NVIDIA, Samsung) has agreed on a standard for cache-coherent memory sharing. It effectively turns the “server” inside out. Instead of memory being inside the server, the server is plugged into a fabric of memory.
The Interplay: How a Packet Lives and Dies
To understand how these three interact, let’s trace a single gradient update in a massive training run:
- Inside the Node (NVLink): The GPU finishes its backward pass. It needs to sync with its 7 neighbors in the rack. They use NVLink and the NVSwitch’s SHARP engine to sum their gradients at 1.8 TB/s.
- Across the Row (InfiniBand): Now, this rack needs to sync with 512 other racks. The data is handed off to an InfiniBand HCA (Host Channel Adapter). Using GPUDirect RDMA, the data moves from the GPU VRAM directly to the IB wire, skipping the system RAM and CPU entirely.
- The Memory Overflow (CXL): During the process, the optimizer states (which track the “momentum” of the learning) are too large for HBM. They are transparently spilled over into a CXL-connected memory expansion module, keeping the pipeline moving without hitting the slow SSD.
Technical Deep Dive: The Convergence of Compute and Networking
Engineers are currently obsessing over the “Unmet Bandwidth Demand.” As GPUs get faster (Blackwell is 5x faster than Hopper in some tasks), the network must also get 5x faster to maintain the same “Compute-to-Comm” ratio.
The Rise of RoCE v2 (The Ethernet Challenger)
We cannot talk about InfiniBand without mentioning RDMA over Converged Ethernet (RoCE). Companies like Meta and Google are pushing RoCE v2 to compete with InfiniBand.
- The Pro: It uses standard Ethernet switches, which are cheaper and have a more robust supply chain.
- The Con: Ethernet is “lossy.” To make RoCE work at scale, you need to implement PFC (Priority Flow Control) and ECN (Explicit Congestion Notification).
- Engineering Reality: Configuring RoCE at the 10,000-node scale is a nightmare of fine-tuning buffers. InfiniBand “just works” out of the box for AI, which is why it commands a premium.
Code Snippet: Setting up the Fabric
When you’re configuring a cluster, your environment variables are where the rubber meets the road. Using the NVIDIA Collective Communications Library (NCCL), you define how these interconnects are used:
# Force NCCL to use InfiniBand for inter-node communication
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3
export NCCL_IB_GID_INDEX=3
export NCCL_IB_RETRY_CNT=7
# Enable NVLink for intra-node communication
export NCCL_P2P_LEVEL=NVL
# Set the maximum number of rings (parallels paths)
# to saturate the 400Gbps/800Gbps links
export NCCL_MAX_NRINGS=16
# Optimize for CXL-based memory offloading (Hypothetical future driver)
export CUDA_MEMORY_POOL_SUPPORT=CXL_SHAREDThe Infrastructure Challenges: The “Nitty-Gritty”
Building these superclusters isn’t just about logic; it’s about physics.
1. The Optical Barrier
At 800Gbps (NDR InfiniBand) and the upcoming 1.6Tbps (XDR), copper cables are essentially useless beyond a few meters. They become too thick, too stiff, and lose too much signal. We are moving to Active Optical Cables (AOCs) and Linear Drive Optics.
- The Engineering Hurdle: Optical transceivers consume a massive amount of power. In a 100k GPU cluster, the networking gear alone can consume several megawatts.
2. The “Incast” Problem
When 1,000 GPUs all send data to a single GPU simultaneously (a common pattern in AI), you get a “Buffer Incast.” If the switch buffer overflows, you get packet loss. In AI, one lost packet = a “Stop the World” event. This is why InfiniBand’s Adaptive Routing is a non-negotiable requirement for high-end LLM training.
3. Thermal Management of the Fabric
NVSwitch chips are now so powerful they require their own liquid cooling manifolds. The Blackwell NVLink switch shelf is a masterpiece of plumbing, managing hundreds of watts of heat just to move data between chips.
Contextualizing the Hype: Is Ethernet Catching Up?
There is immense hype around the Ultra Ethernet Consortium (UEC). Members like AMD, Arista, and Broadcom are trying to “fix” Ethernet for AI.
The Substance: UEC is trying to strip the “cruft” out of the 40-year-old Ethernet protocol. They are adding features like:
- Packet Spraying: Sending different packets of the same message across different paths to avoid hot spots (similar to InfiniBand).
- Flexible Order Delivery: Letting the GPU process packets in whatever order they arrive, rather than waiting for Packet #1 if Packet #2 arrives first.
The Reality: While UEC is promising, InfiniBand and NVLink have a 5-year head start in production-hardened AI environments. For the next two generations of LLMs (GPT-5/6 scale), the “NVIDIA Trinity” (NVLink + IB + HBM) remains the gold standard.
The Engineering Curiosity: Why Not Just One Big Chip?
A common question in engineering circles is: “Why don’t we just make a silicon wafer the size of a pizza and avoid the network?”
This is what Cerebras does with their Wafer-Scale Engine. However, the reason the rest of the industry uses NVLink and InfiniBand is yield and modularity. If a single transistor fails on a pizza-sized chip, the whole thing is trash. With the interconnected approach of NVLink and IB, we can “bin” chips, swap out failed nodes, and scale to sizes that no single piece of silicon could ever reach.
We are essentially using InfiniBand and NVLink to build a Virtual Wafer that spans an entire data center.
Final Thoughts: The Future is Photonic
As we look toward the 200,000-GPU cluster, we are approaching the limit of what electrons moving through copper (or even converted to light at the transceiver) can do.
The next frontier is Silicon Photonics—bringing the fiber optic connection directly onto the GPU die. Imagine an NVLink that doesn’t use electrical traces but instead uses on-chip lasers to communicate with other GPUs at the speed of light.
Until then, the dance between NVLink (the local sprint), InfiniBand (the long-distance marathon), and CXL (the memory warehouse) remains the most sophisticated engineering feat in the modern world. We aren’t just building networks; we are building the substrate for artificial intelligence.
If you’re an engineer in this space, remember: Compute wins the headlines, but the Interconnect wins the convergence. Keep your latencies low and your bandwidth high.