We’ve reached a point in the evolution of artificial intelligence where the bottleneck is no longer the “intelligence” of the algorithm, but the physics of the wire.
If you look at the floor of a modern hyperscale data center—the kind housing 30,000+ H100s or TPU v5ps—you aren’t looking at a collection of individual servers. You are looking at a single, warehouse-scale computer. But this computer is currently choking. As we push toward training models with 10 trillion, 50 trillion, or 100 trillion parameters, we have run head-first into a brutal reality: The “Memory Wall” has been joined by the “Communication Wall.”
In the old world (roughly three years ago), distributed training was about clever sharding and fast Ethernet. In the new world, electrical signaling over copper is becoming a thermal and latency liability. The industry is currently undergoing a violent architectural shift, moving away from traditional distributed GPU clusters toward Photonic Interconnects and Optical Circuit Switching (OCS).
This isn’t just an incremental upgrade. It is a fundamental rewriting of the hardware stack. Let’s dive into why the copper era is ending, how silicon photonics is taking over the rack, and the radical architectures being deployed by the likes of Google, NVIDIA, and Meta to keep the scaling laws alive.
The Brutal Physics of the Status Quo
To understand why we need photonics, we have to understand why copper is failing us. In a massive training run—say, a 1.8-trillion parameter MoE (Mixture of Experts) model—the training process is split across thousands of GPUs using techniques like Data Parallelism, Pipeline Parallelism, and Tensor Parallelism.
In these setups, the GPUs spend a terrifying amount of time waiting. They are waiting for the “AllReduce” or “All-to-All” collective communication operations to finish.
1. The Energy Tax
As we increase the bandwidth of electrical signals (moving from 56G to 112G and now 224G SerDes), the energy required to push those electrons through a copper trace or a Twinax cable increases exponentially. At 224Gbps, the reach of a copper cable is barely a few meters before the signal degrades into noise. To compensate, we use retimers—chips that boost the signal—but these add cost, heat, and, most importantly, nanoseconds of latency.
2. The Radix Problem
In traditional networking, we use a “Leaf-Spine” architecture. If GPU A in Rack 1 needs to talk to GPU B in Rack 50, the packet has to go:
- GPU -> PCIe/NVLink Switch
- NIC (Network Interface Card)
- Top-of-Rack (ToR) Switch
- Spine Switch
- ToR Switch (Destination)
- NIC (Destination)
- GPU (Destination)
Every one of those “hops” involves an O-E-O conversion (Optical-to-Electrical-to-Optical) if using fiber, or multiple stages of electrical switching. Each hop adds ~500ns to 1μs of latency. In a world where HBM3e memory access happens in nanoseconds, spending 10 microseconds on a round-trip network call is an eternity. It destroys the Scaling Efficiency.
Enter the Photonic Fabric: Beyond the Transceiver
For decades, fiber optics were just “dumb pipes” used to connect switches over long distances. We used transceivers (those little SFP/QSFP pluggable modules) to turn electrons into photons for the journey across the data center, then turned them back into electrons as soon as they hit the switch.
Hyperscale AI is moving the optics inside the package.
Silicon Photonics (SiPh) and Co-Packaged Optics (CPO)
The current frontier is Co-Packaged Optics (CPO). Instead of having a separate transceiver module plugged into the front of a switch or server, the optical engine is mounted directly onto the same substrate as the GPU or the Switch ASIC.
Why does this matter?
- Reduced Power: By moving the optical conversion closer to the compute, you eliminate the power-hungry “reach” required to drive signals across a PCB to a pluggable module. You save roughly 30% of total system power.
- Density: You can fit more “lanes” of light into the same area than you can electrical pins. We are talking about Terabits per second per millimeter of die edge.
When you see companies like Ayar Labs or Lightmatter making waves, it’s because they are solving the “IO Bottleneck.” They use microring resonators—tiny circular waveguides that can “drop” or “add” specific wavelengths of light. This allows for Wavelength Division Multiplexing (WDM), where a single fiber carries 8, 16, or 32 different streams of data simultaneously, each on a different “color” of light.
Architecture Deep Dive: Google’s Apollo and the OCS Revolution
While most of the world was focused on InfiniBand, Google quietly built the most advanced AI network on the planet using Optical Circuit Switching (OCS). This is the “Apollo” fabric used for TPU v4 and v5p.
How an OCS Works
A traditional switch (like a Broadcom Tomahawk) is a “Packet Switch.” It looks at every packet, reads the header, and decides where to send it. This is slow and power-hungry.
An Optical Circuit Switch (OCS), like Google’s “Palomar” switch, doesn’t look at packets at all. It uses a grid of MEMS (Micro-Electro-Mechanical Systems) mirrors.
Imagine a grid of thousands of tiny, microscopic mirrors that can tilt on two axes. When Rack A needs to talk to Rack B, the mirrors physically tilt to reflect the laser beam from the input fiber directly into the output fiber.
The Technical Substance:
- Zero Packet Processing: Since the connection is a physical path of light, there is zero switching latency. Once the “circuit” is established, the data moves at the speed of light through glass.
- Dynamic Topology: This is the “killer app.” In a standard cluster, your topology is fixed (e.g., a Fat-Tree). With OCS, you can reconfigure the network topology on the fly via software.
- Resiliency: If a rack of TPUs goes down, the OCS simply “mirrors around” the failure, reconfiguring the entire cluster’s graph in milliseconds to maintain maximum bisection bandwidth.
# Conceptualizing a Dynamic Topology Reconfiguration
def reconfigure_cluster_topology(failed_nodes, ocs_controller):
"""
Real-time adjustment of the physical optical paths
to bypass faulty nodes in a 50k GPU cluster.
"""
current_graph = ocs_controller.get_physical_mapping()
new_graph = compute_optimal_all_to_all_path(current_graph, exclude=failed_nodes)
# Send signals to the MEMS mirrors to tilt
# No packets are dropped; the circuit just changes.
ocs_controller.apply_mirror_angles(new_graph.mirror_vectors)
return "Topology optimized for current health state"The NVLink 5 and Blackwell Breakthrough
At GTC 2024, NVIDIA signaled its own shift toward this photonic future with the Blackwell NVLink Switch.
For years, NVLink was a “within-the-box” protocol. You had 8 GPUs in a HGX baseboard talking to each other at high speeds, but to talk to another box, you had to exit via InfiniBand or Ethernet. This created a “bandwidth drop” (e.g., 900GB/s internal vs. 100GB/s external).
With the GB200 NVL72 architecture, NVIDIA has created a 72-GPU “Single Logical GPU.”
- They use a massive Copper Backplane (because at the rack scale, they’ve managed to optimize copper one last time).
- However, to scale to the 576-GPU pod and beyond, they are integrating LinkX optics.
The Blackwell generation is designed to treat the entire data center as a single fabric. By using the NVLink Switch chip (which has more throughput than the top-of-the-line InfiniBand switches from just two years ago), NVIDIA is essentially building a “Photonic Memory Fabric.”
In this architecture, a GPU can perform a Remote Direct Memory Access (RDMA) to the memory of a GPU three racks away as if it were on its own local PCIe bus. This is the “Disaggregated Data Center.”
Why “Bisection Bandwidth” is the Only Metric That Matters
In the world of AI training, “Peak TFLOPS” is a vanity metric. If you have 100 Petaflops of compute but a network that can only move 10 GB/s, your GPUs will sit idle 90% of the time during the gradient synchronization phase.
This brings us to Bisection Bandwidth: the bandwidth available between two equal halves of the cluster.
The Problem with Ethernet
Standard Ethernet was designed for the internet—where if a packet gets lost, you just re-transmit it (TCP). In AI training, a single lost packet can stall a $100 million training run for milliseconds, which, at scale, costs thousands of dollars in wasted compute time.
This is why we saw the rise of RoCE (RDMA over Converged Ethernet). It tries to make Ethernet act like InfiniBand by allowing GPUs to write directly to each other’s memory without involving the CPU. But even RoCE struggles with congestion management. When 10,000 GPUs all try to send data to the same “Parameter Server” node at once (the “Incast” problem), the buffers in traditional switches overflow.
The Photonic Solution to Congestion
Photonic fabrics, specifically OCS, solve this by being non-blocking by design. Because you aren’t buffering packets in a switch’s memory, you can’t have a buffer overflow. You are simply managing a flow of light.
The Engineering Curiosity: The “Laser” Problem
Here is the “engineering curiosity” that keeps hyperscale architects up at night: Lasers hate heat.
Most silicon chips (CPUs/GPUs) are happy running at 80°C or 90°C. But the Indium Phosphide (InP) lasers used in photonic interconnects are incredibly sensitive to temperature. If the laser is co-packaged next to a 1000W Blackwell GPU, the heat from the GPU will cause the laser’s wavelength to shift, breaking the optical link.
The Solution: Remote Laser Modules (RLM) Architects are now moving the lasers to the front of the rack or even to a separate “laser shelf.”
- A “blind mate” connector feeds the unmodulated light (CW - Continuous Wave) from the cold front of the rack into the hot GPU package.
- The GPU-side silicon photonics chip modulates that light with data.
- The light is then sent back out through the fiber.
This “External Laser” architecture is a marvel of precision engineering. It allows the “brain” (the GPU) to be hot, while the “voice” (the laser) stays cool.
Software-Defined Optics: The New Stack
You can’t just plug in a photonic switch and expect PyTorch to work. The entire collective communication library (like NCCL - NVIDIA Collective Communications Library) has to be rewritten to be topology-aware.
In a standard cluster, NCCL assumes a Ring or Tree topology. It passes data from GPU 0 -> 1 -> 2 -> 3. In a photonic OCS-based cluster, the software says: “Wait, I am doing an All-to-All for a Mixture of Experts model. I should tell the OCS to reconfigure into a Flattened Butterfly topology for the next 50 milliseconds.”
Code Context: The Topology-Aware Scheduler
Modern schedulers (like those used inside Meta’s “Grand Teton” platforms) don’t just allocate “8 GPUs.” They allocate “8 GPUs with the shortest photonic path.”
# Example of a topology-aware resource request (Conceptual)
# Requesting nodes that share a direct OCS-layer circuit
# to minimize 'hops' during tensor-parallel sharding.
srun --nodes=128 \
--network=type=optical,topology=low_latency_mesh \
--gpu-interconnect=nvlink_optical \
--model-sharding=expert_parallel \
python train_10T_model.pyThe Shift to Memory Disaggregation
The ultimate destination of photonic interconnects is Memory Disaggregation.
Right now, if you need more HBM, you have to buy more GPUs. This is inefficient. Photonic fabrics with near-zero latency allow for a “Pool of Memory” architecture. Imagine a rack full of nothing but HBM3e stacks and photonic switches.
Through CXL (Compute Express Link) over Optical, a GPU can “borrow” 100GB of RAM from a different rack with latency low enough that the GPU’s Load/Store units don’t timeout.
This effectively turns the entire data center into a Giant Shared-Memory Machine. This is how we get to 100-trillion parameters. We stop thinking about “nodes” and start thinking about a “fabric of addressable resources.”
The Economic Context: Why Now?
Why did this gain so much attention in the last 12 months?
- The ChatGPT Moment: Suddenly, every company on earth needed to train 100B+ parameter models. The demand for H100s skyrocketed, and people realized that simply buying chips wasn’t enough—they needed to connect them.
- The Power Limit: Data centers are hitting the “Power Wall.” If you can save 20% of your power by switching from copper to light, that’s 20% more GPUs you can fit in the same power envelope. For a 100MW data center, that is a massive competitive advantage.
- The Broadcom/Marvell/NVIDIA Arms Race: The competition in the “AI Switch” market has moved from “Who has the fastest SerDes?” to “Who has the best optical integration?”
The Road Ahead: The “All-Optical” Training Run
We are rapidly approaching the “All-Optical” era of AI. Within the next 2-3 years, we will likely see:
- Fiber-to-the-Chip: Copper will be relegated to power delivery only. All data—even over short distances of 10cm—will be photonic.
- Passive Optical Networks: Using the properties of light (like interference) to perform some parts of the AI calculation inside the network (e.g., Optical All-Reduce).
- Wafer-Scale Interconnects: Extending the concept of the Cerebras Wafer-Scale Engine, but across multiple wafers connected via coherent photonics.
The engineering challenge of our generation isn’t just “more TFLOPS.” It’s “more photons, less heat.” The hyperscalers that master the photonic fabric will be the ones that train the first models capable of true reasoning and multi-modal synthesis at a global scale.
If you’re an engineer today, the message is clear: Learn the physics of light. Because the future of compute is no longer just about switching transistors; it’s about steering lasers.
Technical Glossary for the Modern Architect:
- SerDes (Serializer/Deserializer): The engine that converts parallel data from a chip into high-speed serial bits for transmission.
- PAM4 (Pulse Amplitude Modulation 4-level): The signaling format used to squeeze 2 bits of data into every clock cycle (doubling the bandwidth of older NRZ methods).
- Waveguide: A physical structure (like a silicon “wire”) that guides light on a chip.
- BER (Bit Error Rate): The frequency of errors in a stream; as we go faster, we rely more on FEC (Forward Error Correction) to fix the “noisy” reality of high-speed signaling.
- InP (Indium Phosphide): The material usually used to make the lasers themselves, as silicon cannot efficiently emit light.