We’ve reached a strange inflection point in the history of computing. For decades, the mantra was "more FLOPs." We worshipped at the altar of raw floating-point operations per second, pushing silicon to crunch numbers faster and faster. But if you walk into the engineering labs of OpenAI, Anthropic, or Meta today, you won’t hear them complaining about a lack of compute.
They’re complaining about the plumbing.
In the era of trillion-parameter Large Language Models (LLMs), the GPU has ceased to be the bottleneck. The real war is being fought in the copper and glass that connects them. We are no longer limited by how fast we can multiply matrices; we are limited by how fast we can move weights from HBM (High Bandwidth Memory) to the registers, and how fast we can synchronize those states across a massive, sprawling fabric of thousands of GPUs.
If you’re trying to run inference on a 1.8-trillion parameter Mixture-of-Experts (MoE) model with sub-second latency, you aren't just an AI engineer—you’re a distributed systems architect fighting the laws of physics. Let’s decode why the interconnect is the new frontier and how custom fabric solutions are the only way out of the memory bottleneck.
The Brutal Physics of the Memory Wall
To understand why interconnects matter, we have to talk about Arithmetic Intensity. This is the ratio of total floating-point operations (FLOPs) performed to the total bytes of data moved from memory.
In the "Training" phase, life is relatively good. We use large batch sizes, which means we load a weight once and use it for many different tokens in a batch. The arithmetic intensity is high. The compute cores stay busy.
In the "Inference" (Decoding) phase, the world falls apart. During the autoregressive generation of a single token, we have to read every single weight of that trillion-parameter model just to produce one word. If your model is 1.8 trillion parameters in FP16, that’s 3.6 Terabytes of data that must be moved from VRAM to the GPU’s registers just to generate "the" or "and."
The Math of Despair
Let’s look at a modern H100 GPU. It has a peak HBM3 bandwidth of roughly 3.35 TB/s.
- Model Size: 1.8T parameters (FP16) = 3.6 TB.
- Theoretical Min Latency for 1 Token: 3.6 TB / 3.35 TB/s ≈ 1.07 seconds per token.
A human reads at about 5-10 tokens per second. Our "state-of-the-art" single GPU is already over an order of magnitude too slow for a trillion-parameter model, and that’s assuming 100% memory efficiency (which never happens). This is the Memory Wall. To scale, we have no choice but to slice the model across multiple GPUs (Tensor Parallelism) or multiple nodes (Pipeline Parallelism).
The moment you slice the model, you move the bottleneck from the internal memory bus to the interconnect.
The Hierarchy of Interconnects: From NVLink to the World
When we scale a model across 128 or 512 GPUs, we create a hierarchy of communication. Each level of this hierarchy has vastly different latency and bandwidth profiles.
1. The Intra-Node Speed Demon: NVLink
Within a single "box" (like a DGX H100), we use NVLink. Think of this as a private, high-speed highway that bypasses the sluggish PCIe bus. NVLink 4.0 provides 900 GB/s of bidirectional bandwidth. This is where we perform Tensor Parallelism (TP).
In TP, we split a single weight matrix across 8 GPUs. During every layer of the transformer, the GPUs must perform an All-Reduce operation to synchronize their results. Because this happens multiple times per layer, even a few microseconds of latency will cause the GPU cores to "starve," sitting idle while they wait for their neighbors.
2. The Inter-Node Backbone: InfiniBand vs. RoCE
Once you grow beyond 8 GPUs, you’re leaving the chassis. Now you’re dealing with cables.
- InfiniBand (IB): The gold standard for HPC. It’s a lossless, credit-based network with incredibly low latency (sub-1 microsecond). It handles the heavy lifting of
All-To-Allcommunications in MoE models. - RoCE v2 (RDMA over Converged Ethernet): The "cloud-scale" alternative. It’s cheaper and runs on Ethernet, but it’s lossy by nature and requires complex congestion control (like DCQCN) to prevent performance collapses at scale.
The industry is currently in a heated debate: Can we build a fabric large enough for a 10-trillion parameter model using standard Ethernet? Or is the "jitter" (latency variance) of Ethernet too destructive for synchronous AI workloads?
The MoE Crisis: Why Mixture-of-Experts Broke the Network
The industry has pivoted toward Mixture-of-Experts (MoE) architectures (like GPT-4 or Grok-1) to cheat the FLOPs-to-Parameter ratio. Instead of one giant dense model, you have a router that sends data to only 2 out of 16 "experts" per layer.
This is great for compute efficiency, but it is an interconnect nightmare.
In a dense model, communication is predictable. In an MoE model, where different experts live on different nodes, we have to perform All-to-All communication.
- The Problem: Every GPU needs to send a different piece of data to every other GPU in the cluster.
- The Result: "Incaster" congestion. If 64 GPUs all try to send data to the one GPU that holds "Expert #7" at the same time, the buffers overflow, packets drop, and the entire inference pipeline grinds to a halt.
This is why "Custom Fabric" is no longer a buzzword—it's a survival requirement.
Custom Fabric Solutions: Beyond "Off-the-Shelf"
To solve the All-to-All bottleneck, companies are moving away from traditional leaf-spine network topologies and toward custom, tightly integrated fabrics.
NVIDIA’s NVL72: The "Giant GPU" Approach
NVIDIA’s recently announced Blackwell NVL72 architecture is a masterclass in custom fabric engineering. They’ve essentially turned an entire rack of 72 GPUs into a single logical unit.
- The NVSwitch Fabric: Instead of just connecting GPUs, the entire rack is built around a massive 130 TB/s internal backplane.
- The Copper Revolution: They moved from optical cables to 2 miles of internal copper wiring for the NVLink Switch. Why? Because it saves 20kW of power that would have been wasted on optical-to-electrical conversion, and it cuts latency to the bone.
- Result: All 72 GPUs can talk to each other as if they were on the same board. For a trillion-parameter MoE model, this means "Expert Parallelism" can happen at rack-scale with almost no penalty.
Google’s TPU v5p and the Optical Circuit Switch (OCS)
Google took a different route. Instead of massive copper backplanes, they built their own Optical Circuit Switches (OCS). Traditional switches convert light to electricity, route it, and convert it back to light. Google’s OCS uses MEMS (Micro-Electro-Mechanical Systems) mirrors to physically tilt and reflect light beams.
- The Advantage: Near-zero latency and massive power savings.
- The Flexibility: They can dynamically reconfigure the topology of their 8,960-TPU pods to match the specific "shape" of the model’s communication pattern (e.g., changing from a 3D Torus to a Twisted Torus on the fly).
Software to the Rescue: Overlapping and Paging
Hardware can’t solve everything. Even with a custom fabric, we still have to manage the "Tail Latency" (the 99.9th percentile of delay). This is where the engineering of the Inference Engine (like vLLM or TensorRT-LLM) becomes critical.
1. PagedAttention
One of the biggest silent killers of inference performance is the KV Cache. As an LLM generates text, it stores the "keys" and "values" of previous tokens in memory. For a trillion-parameter model with a long context window, this cache can take up hundreds of gigabytes. Inspired by traditional OS virtual memory, PagedAttention allows the KV cache to be stored in non-contiguous memory blocks. This prevents fragmentation and allows for "Continuous Batching," ensuring the fabric is always saturated and never waiting for a "straggler" GPU to finish a large block.
2. Communication-Compute Overlapping
The goal of any high-performance kernel is to hide the interconnect latency. While the GPU is crunching the numbers for Layer $N$, the fabric should already be pre-fetching the weights for Layer $N+1$.
# Conceptual pseudocode for overlapping
with stream(compute_stream):
run_matrix_multiply(layer_i)
with stream(communication_stream):
# This happens in parallel with the compute above
all_gather_weights(layer_i + 1)
In modern trillion-parameter stacks, we use Custom Collectives. Instead of using generic NCCL (NVIDIA Collective Communications Library) calls, engineers write hand-optimized kernels that shard the data and use "pipelined" All-Reduce, where parts of the data are transmitted while other parts are still being computed.
The Emerging Challenge: The Prefill vs. Decode Divergence
We are now seeing a split in infrastructure requirements for the two stages of inference:
- The Prefill Phase: The user sends a 10,000-word prompt. The model processes it all at once. This is compute-bound. It loves high FLOPs and large batch sizes.
- The Decode Phase: The model generates one token at a time. This is memory-bound and interconnect-bound. It loves high bandwidth and low latency.
The industry is moving toward Disaggregated Prefill and Decode. In this architecture, you have one cluster of GPUs optimized for "The Crunch" (Prefill) and another cluster optimized for "The Plumbing" (Decode). They share the KV Cache across a massive, ultra-low-latency CXL (Compute Express Link) or InfiniBand fabric.
This prevents the "Decode" GPUs from being held hostage by a new, massive incoming request, keeping your tokens-per-second high and consistent.
The Future: Silicon Photonics and the Death of the Motherboard
As we look toward 10-trillion and 100-trillion parameter models, even the most advanced copper fabrics will fail. The electrical resistance over copper creates too much heat, and the signal integrity degrades at the 224 Gbps-per-lane speeds we are approaching.
The "End Game" for interconnects is Silicon Photonics. Imagine replacing the electrical SerDes on a GPU with a laser. Instead of sending electrons through a copper trace on a PCB, the GPU directly emits light into a fiber optic cable. This isn't just about speed; it's about disaggregation.
With silicon photonics, the distance between the compute and the memory stops being a constraint. You could have a rack of "Compute Trays" and a rack of "HBM Trays" connected by a transparent optical fabric. The "Memory Wall" is effectively demolished because you can scale your memory capacity and bandwidth independently of your GPU count.
Why This Matters for the Rest of Us
You might be thinking, "I’m not building a trillion-parameter model; why should I care about rack-scale fabrics?"
Because the engineering breakthroughs happening at the "Trillion-Scale" are trickling down. The techniques used to optimize RoCE v2 for 512-GPU clusters are making standard data center networking more resilient. The innovations in PagedAttention are making it possible to run 70B models on a single consumer workstation.
But more importantly, we are shifting our definition of "The Computer." In the 1990s, the computer was the CPU. In the 2010s, it was the GPU. In the 2020s, the cluster is the computer.
The interconnect isn't just a cable anymore; it's the system bus of the world’s largest AI brains. If you want to understand where AI is going, stop looking at the TFLOPS on the spec sheet and start looking at the bandwidth and latency of the fabric.
The "Intelligence" isn't just in the weights—it's in the way the weights move.
Key Takeaways for the Infrastructure Engineer:
- Inference is a Memory Problem: In the decoding phase, bandwidth is king, not FLOPs.
- The Interconnect Hierarchy is Vital: Understand the transition from NVLink (intra-node) to InfiniBand/RoCE (inter-node).
- MoE Scales with Fabric: Mixture-of-Experts performance is directly tied to your network's ability to handle All-to-All congestion.
- Topology Matters: Moving toward rack-scale "logical GPUs" (like NVL72) is the only way to minimize the "Latency Tax" of distributed inference.