For the last forty years, the basic blueprint of a computer has remained stubbornly static: a motherboard, some CPUs, and a fixed amount of RAM plugged into slots just inches away. In the world of hyperscale cloud providers—the Googles, AWSs, and Azures of the world—this “pizza box” architecture has become a multi-billion dollar liability.
If you’ve ever managed a large-scale cluster, you’ve lived the nightmare of Stranded Memory. You have one node running a compute-heavy job with 90% CPU load but using only 10% of its RAM. Next to it, another node is starving for memory, but its CPU is idling. Because that RAM is physically trapped behind a DDR bus on a specific motherboard, it might as well be on the moon.
Estimates from Microsoft Azure suggest that up to 25% of all DRAM in their data centers is “stranded” at any given time. In an industry where memory accounts for nearly 40-50% of the total server BOM (Bill of Materials) cost, that is an astronomical amount of capital rotting in the rack.
We are currently witnessing a tectonic shift in infrastructure. We are moving away from “servers” and toward Disaggregated Architectures. By leveraging CXL (Compute Express Link) and Silicon Photonics, we are finally decoupling memory from the CPU, allowing us to treat an entire data center rack as a single, giant, fluid pool of resources.
The Memory Wall and the Death of Copper
To understand why we need CXL and Photonics, we have to look at the physics of the “Memory Wall.”
Historically, we scaled performance by increasing CPU core counts. However, the number of memory channels a CPU can support is limited by physical pin-out constraints and power density. As we moved from DDR4 to DDR5, we hit a wall. Adding more DIMM slots increases trace length, which introduces signal integrity issues and latency.
Furthermore, we are reaching the limits of copper interconnects. At the speeds required for PCIe 5.0 (32 GT/s) and PCIe 6.0 (64 GT/s), electrical signals in copper wires degrade incredibly fast. You can only move data a few inches before you need “retimers”—expensive, power-hungry chips that clean up the signal.
This is the bottleneck. We need a way to move memory further away from the CPU without killing performance, and we need a protocol that allows the CPU to treat that distant memory as “local.”
Enter CXL: The Protocol That Changed Everything
CXL (Compute Express Link) is an open industry standard built on top of the PCIe physical layer. But while PCIe is a “discovery and configuration” protocol designed for peripherals, CXL is a coherency protocol.
It allows the CPU and an external device (like a memory expansion buffer or a GPU) to share a single memory space with extremely low latency. CXL operates using three distinct protocols:
- CXL.io: Based on PCIe, used for device discovery, configuration, and interrupts.
- CXL.cache: Allows a peripheral to cache CPU memory with high efficiency.
- CXL.mem: This is the crown jewel. It allows the CPU to access memory on a peripheral device using standard Load/Store instructions.
The Three Types of CXL Devices
In a disaggregated world, we categorize hardware by how it interacts with these protocols:
- Type 1 (Accelerators without local memory): Think of smart NICs. They use CXL.io and CXL.cache to track CPU memory.
- Type 2 (Accelerators with local memory): Think of GPUs or FPGAs. They have their own HBM (High Bandwidth Memory) but use CXL to create a unified, coherent memory space between the CPU and the GPU.
- Type 3 (Memory Expanders): These are the heroes of disaggregation. A Type 3 device is essentially a “box of RAM” that plugs into a CXL port. To the OS, this looks like just another NUMA (Non-Unified Memory Access) node.
CXL 2.0 and 3.0: The Fabric Revolution
While CXL 1.1 allowed for simple point-to-point memory expansion, CXL 2.0 introduced switching. This allows a single pool of memory to be “partitioned” and assigned to different servers dynamically.
CXL 3.0 (based on PCIe 6.0) takes this to the extreme. It introduces Fabric capabilities, supporting multi-level switching, spine-leaf topologies, and—most importantly—memory sharing. In CXL 2.0, a chunk of memory could be assigned to Server A or Server B. In CXL 3.0, both Server A and Server B can access the same memory simultaneously with hardware-level coherency.
The Physical Transport: Why Silicon Photonics is Non-Negotiable
You can have the best protocol in the world, but if you’re limited by the 10-centimeter reach of a copper PCIe trace, you can’t build a disaggregated data center. You can only build a slightly more flexible server.
This is where Silicon Photonics (SiPh) enters the fray.
Photonics replaces electrons moving through copper with photons moving through glass (fiber). Historically, optical transceivers were bulky “pluggable” modules (like QSFP28) used for networking between switches. They were too big, too power-hungry, and too expensive to use for CPU-to-Memory links.
Co-Packaged Optics (CPO)
The industry is moving toward Co-Packaged Optics. Instead of having a separate transceiver module, we are integrating the optical engines directly onto the CPU or CXL switch package.
By bringing the laser and the modulator millimeters away from the silicon die, we eliminate the need for power-hungry electrical drivers. This allows us to achieve:
- Massive Bandwidth Density: Terabits per second per millimeter of die edge.
- Ultra-low Latency: Light travels through fiber with minimal degradation, allowing us to put memory pools meters (or even tens of meters) away from the CPU while maintaining the nanosecond-scale latencies required for load/store operations.
Imagine a rack where the “Backplane” isn’t a copper PCB, but an Optical Circuit Switch (OCS). Any CPU in the rack can connect to any GPU or Memory appliance via a burst of light, bypassing the traditional bottleneck of the Top-of-Rack (ToR) switch.
Architectural Deep Dive: The Disaggregated Rack
Let’s look at how this actually looks in a modern hyperscale design.
In a traditional setup, you have 40 independent servers. In a Disaggregated CXL-over-Photonics rack, you have:
- Compute Sleds: These are “thin” nodes containing only CPUs and a small amount of “Type 0” local RAM for the kernel to boot.
- Memory Appliances: 2U chassis filled with nothing but E3.S CXL memory modules (think 16TB to 64TB of RAM per appliance).
- Accelerator Pools: Trays of GPUs or TPUs connected via CXL.
- The Photonics Fabric: A low-latency optical interconnect that stitches them together.
The Software Challenge: NUMA on Steroids
From a systems engineering perspective, disaggregated memory isn’t “free.” It introduces a tiered memory hierarchy.
- Tier 0: On-die cache (L1/L2/L3).
- Tier 1: Local DDR5 RAM (Directly attached).
- Tier 2: CXL-attached memory (Same rack, <100ns additional latency).
- Tier 3: Pooled CXL memory (Across a switch, <250ns additional latency).
The Linux kernel needs to be incredibly smart about this. This is why there has been a flurry of activity in the Tiered Memory Management (TMM) and Memory Demotion subsystems of the kernel.
The goal is to use “Hot/Cold” page tracking. The kernel keeps “hot” frequently accessed data in Tier 1 (local RAM) and “demotes” cold pages to Tier 2/3 (CXL memory). If a process suddenly needs a cold page, the hardware/kernel must “promote” it back to local RAM or access it over the CXL bus with minimal stall.
// Simplified logic for a hypothetical Tiered Memory Allocator
void* allocate_memory(size_t size, int priority) {
void* ptr = NULL;
if (priority == HIGH_PERFORMANCE) {
ptr = numa_alloc_onnode(size, LOCAL_DDR_NODE);
}
if (!ptr) {
// Fallback to CXL Pooled Memory
ptr = numa_alloc_onnode(size, CXL_POOL_NODE);
mark_page_as_external(ptr);
}
return ptr;
}The “Why Now”: The Generative AI Explosion
If disaggregation is so hard, why are we doing it now? Two words: Large Models.
Training an LLM like GPT-4 requires thousands of GPUs, but inference and fine-tuning are increasingly memory-bound. When you’re running a model with 1.8 trillion parameters, the KV (Key-Value) cache for long-context windows can consume terabytes of RAM.
Building a single server with 2TB of local DDR5 is physically impossible or prohibitively expensive. But with CXL and Photonics, an AI inference engine can “borrow” 2TB of RAM from a pooled memory appliance, process the request, and then release that memory back to the pool for a different job.
This is “Composable Infrastructure.” You are no longer buying a server; you are renting a slice of a giant, rack-scale computer.
The Engineering Curiosities: Dealing with “Far” Memory
One of the most fascinating technical hurdles in disaggregated memory is Error Handling.
In a traditional server, if a DIMM fails, the machine crashes (MCE - Machine Check Exception). In a disaggregated world, the “memory” is at the other end of an optical cable. What happens if someone unplugs the cable?
We can’t just let the CPU hang indefinitely waiting for a Load instruction that will never return. CXL addresses this with complex Timeout and Poison mechanisms. If the CXL link goes down, the fabric must notify the CPU immediately so the kernel can “SIGBUS” the affected processes rather than kernel-panicking the entire machine.
Furthermore, we are seeing the rise of Memory Side Caches. To hide the 100-200ns latency of pooled memory, CXL controllers often include a small amount of high-speed SRAM or a small local DDR buffer to act as a cache for the “far” memory.
The Economic and Environmental Impact
Beyond the technical wizardry, there’s a massive sustainability angle.
- Reduced E-Waste: Today, when a CPU becomes obsolete, we often toss the whole server—RAM and all. In a disaggregated data center, you can upgrade the “Compute Sleds” while keeping the “Memory Appliances” for another two generations.
- Power Efficiency: By eliminating hundreds of tiny voltage regulator modules (VRMs) on individual motherboards and consolidating them into efficient, high-density power shelves for memory pools, we significantly reduce conversion losses.
- Utilization: If we solve the “Stranded Memory” problem, we can increase data center density by 20-30% without building a single new square foot of floor space.
The Road Ahead: 2025 and Beyond
We are currently in the “Early Adopter” phase. Intel’s Sapphire Rapids/Emerald Rapids and AMD’s Genoa/Bergamo processors are the first to truly support CXL 1.1/2.0 at scale.
The next three years will be the era of the CXL Switch. Companies like Astera Labs, Marvell, and Credo are racing to build the “Cisco of CXL”—the silicon that will route memory requests across the rack.
As Silicon Photonics moves from specialized lab equipment to high-volume manufacturing, the physical barriers to disaggregation will vanish. We will stop talking about “servers” and start talking about “Resource Domains.”
The dream of the “Data Center as a Computer” is finally becoming a physical reality. It’s not just a software abstraction anymore—it’s a coherent, light-speed fabric of silicon and glass.
Are you ready for the age of pooled memory? If you’re a systems programmer, it’s time to start thinking about NUMA distances in terms of light-nanoseconds. The wall is coming down, and the pool is open.