The AI industry is currently obsessed with a single metric: FLOPS. We talk about Teraflops and Petaflops as if they are the sole currency of intelligence. But if you talk to the engineers at Google, NVIDIA, or Meta who are actually building the clusters that train models like Gemini 1.5 Pro or GPT-4, they’ll tell you a different story. They don’t care about how fast the math is; they care about how fast the data moves.
We have hit the “Memory Wall.” Our ability to perform matrix multiplication has far outpaced our ability to feed those multipliers with data. This bottleneck has forced a radical architectural pivot. We are no longer building faster chips; we are building faster fabrics.
Today, we’re dissecting the silicon-level sorcery that makes this possible. We’re looking at the Intra-Node Fabric, the emergence of NVLink-C2C as a gold standard, and how Google’s TPU v6 (Trillium) is fundamentally decoupling compute from memory to create a planetary-scale supercomputer.
The Death of the Monolith: Why C2C is the New King
For decades, the “node” was the atomic unit of computing. You had a CPU, some RAM, and maybe a few GPUs connected via PCIe. But PCIe is, frankly, a relic in the age of LLMs. Even PCIe Gen 5, with its 128 GB/s bi-directional bandwidth, is a straw trying to empty an ocean.
When training a model with trillions of parameters, the weights and gradients are spread across thousands of chips. If a chip has to wait 10 milliseconds for a packet to traverse a traditional network stack, the expensive HBM (High Bandwidth Memory) sits idle. This is why Chip-to-Chip (C2C) interconnects have become the most critical piece of real estate in the data center.
The NVLink-C2C Paradigm
NVIDIA’s NVLink-C2C (Chip-to-Chip) is the industry’s benchmark for this. It’s a low-latency, high-bandwidth, coherent interconnect that allows silicon die—like a Grace CPU and a Blackwell GPU—to talk to each other as if they were on the same piece of silicon.
What makes C2C different from standard networking?
- Coherency: The CPU and GPU share a unified memory space. There is no “copying” data from RAM to VRAM. The GPU simply reaches into the CPU’s LPDDR5X memory with near-native latency.
- Energy Efficiency: Moving a bit of data across a PCB via C2C consumes orders of magnitude less energy (picojoules per bit) than sending it across an Ethernet cable.
- Bandwidth Density: We’re talking about 900 GB/s of bi-directional bandwidth in a single link.
But while NVIDIA is dominating the commercial market with NVLink, Google has been quietly perfecting a different, perhaps more radical, approach with their TPU v6 Trillium pods.
Inside Trillium: Google’s TPU v6 Architecture Deep-Dive
Google’s Trillium (TPU v6) isn’t just an incremental update; it’s a redesign of how compute nodes interact with the world. While NVIDIA focuses on the GPU-to-CPU bond, Google is obsessed with the Pod-scale fabric.
1. The ICI (Inter-Connect Interface) vs. NVLink
In a TPU v6 Pod, the “fabric” isn’t an afterthought—it’s the backbone. Google uses a proprietary ICI (Inter-Connect Interface). In Trillium, the ICI bandwidth has been boosted by a staggering 4.7x per chip compared to TPU v5e.
Each Trillium chip is connected in a 3D Torus topology. This means every chip has direct, high-speed “pipes” to its neighbors in three dimensions. Why does this matter? Because in LLM training, we use Ring All-Reduce algorithms. If a chip can talk to its neighbors at terabit speeds without hitting a top-of-rack switch, the “effective” compute power of the cluster scales linearly rather than logarithmically.
2. The Decoupling: Sparse Cores and Memory Disaggregation
One of the most technical “under-the-hood” shifts in TPU v6 is the transition to a more decoupled memory architecture.
Traditionally, if you needed more HBM, you had to buy more compute (more TPUs). Trillium pushes the boundaries of memory disaggregation. By using ultra-low latency intra-node fabrics, Google can essentially pool HBM resources.
- The Sparse Core: TPU v6 features enhanced “Sparse Cores.” These are specialized units designed to handle the embedding tables and sparse operations common in recommendation systems and MoE (Mixture of Experts) models.
- The Fabric Benefit: By decoupling the Sparse Core from the dense Tensor Core, Google can scale the memory-intensive parts of a model independently of the compute-intensive parts. The intra-node fabric acts as the “glue” that allows a Tensor Core on Chip A to access an Embedding Table on Chip B with almost zero penalty.
The Physics of Scale: 65,536 TPUs in a Single Fabric
The headline figure for Trillium is its ability to scale to tens of thousands of chips in a single cluster. To put this in perspective, a single TPU v6 Pod can deliver more aggregate FLOPS than the world’s top supercomputers from just five years ago.
But how do you keep 65,536 chips synchronized? You can’t use copper. Copper has a “reach” problem—at high frequencies, signal degradation is so bad you can only move data a few centimeters.
The Optical Circuit Switch (OCS)
Google’s secret weapon in the TPU v6 Pod is the OCS. Unlike traditional packet switches (which convert light to electricity, process the packet, and convert back to light), the OCS uses MEMS (Micro-Electro-Mechanical Systems) mirrors.
- Zero Latency Switching: The OCS physically tilts tiny mirrors to route laser beams between TPU racks. There is no electrical processing in the middle.
- Dynamic Topology: If a rack of TPUs fails, the OCS can “re-patch” the entire data center’s topology in milliseconds by simply moving mirrors. This is the ultimate expression of decoupling compute from the physical network.
Hardware-Software Co-Design: The Role of XLA
You can have the fastest fabric in the world, but if your software doesn’t know how to use it, it’s just expensive space heating. This is where XLA (Accelerated Linear Algebra) comes in.
XLA is the compiler that powers JAX and TensorFlow. For TPU v6, XLA has been updated to be “fabric-aware.” It doesn’t just compile code for a single chip; it compiles for the entire Pod.
# A conceptual look at how XLA handles the fabric
# The compiler automatically shards the tensor across the ICI fabric
import jax
import jax.numpy as jnp
from jax.sharding import Mesh, PartitionSpec, NamedSharding
# Define the physical mesh of TPU v6 chips
devices = jax.devices()
mesh = Mesh(devices.reshape((8, 8)), axis_names=('x', 'y'))
# The fabric allows us to treat 64 chips as one giant memory space
sharding = NamedSharding(mesh, PartitionSpec('x', 'y'))
@jax.jit
def train_step(weights, data):
# XLA handles the underlying ICI/C2C communication
# to ensure 'weights' are synchronized across the 3D Torus
logits = jnp.dot(data, weights)
return logitsIn the snippet above, the engineer doesn’t have to write “Send packet to TPU #42.” The Intra-Node Fabric and the XLA compiler treat the cluster as a single, distributed device. The NVLink-C2C or ICI handles the “load/store” operations at the hardware level, making distributed computing feel like local computing.
The Hype vs. The Substance: Why “C2C” is the Final Frontier
There is immense hype around “Unified Memory” and “Superchips.” You’ve likely seen the benchmarks showing NVIDIA Blackwell crushing previous records. But the substance behind the hype is a fundamental shift in data center economics.
The Power Wall
We are reaching the limits of power delivery. A single modern AI rack can pull over 100kW. A significant portion of that power is wasted just moving data across long copper traces. By moving to C2C and Optical Fabrics, we are significantly reducing the pJ/bit (picojoules per bit).
Reliability at Scale
When you have 50,000 TPUs, something is always breaking. In a traditional rigid architecture, one dead chip could take down a whole training job. Because Google has decoupled the compute from the network via the OCS and the ICI fabric, they can “route around” failures without stopping the world. This is the difference between a 70% “Goodput” and a 95% “Goodput” in model training.
Dissecting the “Decoupling”
What does it actually mean to “decouple compute from memory”? In the context of TPU v6, it represents a three-layer abstraction:
- Physical Decoupling: Through ICI and OCS, the physical location of a chip in a rack matters less. We are moving toward a “pool of chips” rather than a “server of chips.”
- Logical Decoupling: The HBM3e on a Trillium chip is accessible by its neighbors with such low latency that the L3 cache of the cluster is effectively the HBM of the adjacent nodes.
- Operational Decoupling: Google can upgrade the networking (OCS) independently of the compute (TPU), or vice-versa.
The Performance Gains
The result of this intra-node fabric evolution in TPU v6 is staggering:
- MLPerf Benchmarks: Trillium shows up to a 67% increase in energy efficiency over v5p.
- Training Time: For massive LLMs, the bottleneck is usually the “All-Reduce” step. By doubling the ICI bandwidth and using C2C-style low-latency paths, Google has effectively cut the communication overhead of large-scale training by half.
The Engineering Curiosity: How do you cool a Fabric?
One detail often overlooked in engineering blogs is the thermal cost of connectivity. High-speed SerDes (Serializer/Deserializer)—the circuits that drive these C2C and ICI links—run incredibly hot.
In TPU v6, Google utilizes direct-to-chip liquid cooling. But it’s not just for the TPU cores. The cooling plates are designed to cover the ICI high-speed transceivers. If those transceivers throttle due to heat, the entire Pod slows down to the speed of the slowest link. The fabric is a living, breathing entity that requires as much thermal management as the AI processors themselves.
The Future: Will we ever go back?
The era of the “standalone server” is over for AI. Whether it’s NVIDIA’s NVLink-C2C creating a seamless bond between CPU and GPU, or Google’s TPU v6 Trillium creating a continental-scale optical fabric, the message is clear: The network is the computer.
We are moving toward a future where “memory” isn’t something that lives on a stick next to your CPU, but a shared resource that exists across a high-speed, coherent fabric. The decoupling we see in Google’s Pods is just the beginning.
As we look toward TPU v7 and the next generation of NVIDIA Rubin, the “Intra-Node Fabric” will likely transition entirely to Silicon Photonics. We will stop using electrons to move data between chips and start using light on the die itself. When that happens, the “Memory Wall” won’t just be scaled—it will be demolished.
Key Takeaways for the Modern Engineer:
- Bandwidth is the new FLOPS: When evaluating AI infrastructure, look at the C2C/ICI bandwidth, not just the TFLOPS.
- The Fabric defines the Scaling: A 3D Torus or a Fat-Tree topology with Optical Switching determines your “Goodput” more than the raw speed of a single chip.
- Software must be Fabric-Aware: Compilers like XLA are no longer optional; they are the interface to the supercomputer.
The next time you see a “Trillium” or “Blackwell” announcement, don’t just look at the chip. Look at the wires. That’s where the real magic is happening.