If you’ve spent any time lately monitoring a fleet of H100s or A100s during a large-scale LLM training run, you’ve likely stared at a dashboard that feels like a personal insult. Your GPUs are pegged at 95% utilization, yet your job is crashing with the most dreaded acronym in modern engineering: OOM (Out of Memory).
We are currently living through the “Memory Wall” era. While GPU compute throughput (FLOPS) has increased by roughly 1000x over the last decade, memory bandwidth and capacity have only crawled forward by about 30x. We’ve been trying to solve this by throwing more HBM (High Bandwidth Memory) at the problem, but HBM is expensive, physically limited by the size of the interposer, and—perhaps most frustratingly—it is rigid.
In a typical AI supercluster today, memory is “stranded.” Node A might be starving for memory while Node B has 64GB of idle capacity it can’t share. This inefficiency is the silent killer of TCO (Total Cost of Ownership) in the data center.
Enter CXL (Compute Express Link).
What started as a niche interconnect specification has evolved into the architectural backbone of the next generation of AI infrastructure. CXL isn’t just another cable; it’s a fundamental shift in how we think about the “box.” We are moving away from a world of monolithic servers and toward a world of disaggregated, composable infrastructure.
In this deep dive, we’re going to look under the hood of CXL 2.0 and 3.0, explore how memory pooling is solving the “stranded memory” problem, and analyze why the future of AI scaling depends more on the fabric than the silicon.
The Anatomy of the Problem: Why HBM is Not Enough
Before we dive into CXL, we have to understand why we’re in this mess. In a GPU-centric world, HBM is king because it’s physically stacked on the GPU package, providing terabytes per second of bandwidth. But HBM has three fatal flaws for the scale of models we are building (think 10T+ parameters):
- Capacity Ceilings: You can’t just “add more RAM” to an H100. You are limited by what’s on the package.
- The “Stranded” Problem: In a cluster of 10,000 GPUs, if every GPU has 80GB, you have 800TB of memory. But if a specific layer of your model needs 90GB on one node, you can’t “borrow” 10GB from the neighbor. You have to scale the entire cluster or shard the model further, which introduces massive communication overhead.
- Blast Radius and Cost: HBM is significantly more expensive per GB than standard DDR5. Using HBM for “cold” data (like optimizer states or KV caches that aren’t being immediately accessed) is an architectural sin.
This is where CXL steps in. It provides a way to connect GPUs and CPUs to a massive, external pool of DDR5 memory at load/store latencies, effectively turning the entire rack into a single, giant memory space.
What is CXL, Really? (The Technical Substance)
At its physical layer, CXL runs on the PCIe Gen5 (and Gen6 for CXL 3.0) electrical interface. But CXL is more than just PCIe with a fancy name. While PCIe is a non-deterministic, producer-consumer protocol (great for storage, bad for memory), CXL is a cache-coherent, low-latency protocol.
CXL is comprised of three distinct protocols that run on the same physical link:
- CXL.io: Think of this as the “management plane.” It handles discovery, configuration, and register access. It’s essentially PCIe with some enhancements.
- CXL.cache: This allows a peripheral device (like a GPU or FPGA) to access and cache memory from the host CPU with extremely low latency.
- CXL.mem (The Holy Grail): This allows the host CPU or other accelerators to access a device’s memory as if it were local system RAM. This is what enables memory pooling.
The Latency Math
To appreciate CXL, you have to look at the numbers.
- Local DRAM access: ~100ns.
- CXL-attached memory access: ~170ns to 200ns.
- RDMA (Remote Direct Memory Access) over Ethernet/InfiniBand: 1,000ns to 5,000ns+.
CXL sits in the “Goldilocks zone.” It’s slightly slower than local RAM, but it’s fast enough that the CPU/GPU can perform load/store instructions directly to it without the massive software stack overhead of a traditional network call.
Memory Pooling vs. Disaggregation: The Architectural Shift
When we talk about the role of CXL in AI superclusters, we are usually talking about two specific concepts: Pooling and Disaggregation.
1. Memory Pooling (CXL 2.0)
In CXL 2.0, we introduced the CXL Switch. This allows multiple “hosts” (servers) to connect to a single “device” (a memory expansion drawer).
Imagine a rack with 8 GPU nodes and a single CXL memory appliance containing 4TB of DDR5. The CXL switch can dynamically allocate 512GB to Node 1, 1TB to Node 2, and so on. If Node 1 finishes its job, that memory can be re-assigned to Node 3 in milliseconds without a reboot.
The Engineering Win: You no longer need to over-provision every server for its “peak” memory usage. You provision for the average usage and use the CXL pool to handle the spikes.
2. Full Disaggregation (CXL 3.0)
CXL 3.0 (built on PCIe Gen6) is where things get truly wild. It introduces Fabric capabilities.
In CXL 2.0, communication was still mostly “tree-like” (host to switch to device). CXL 3.0 allows for peer-to-peer communication across a fabric of up to 4,096 nodes.
- Port-based routing: Allows for complex topologies (Spine-Leaf) just like a standard data center network.
- Memory Sharing: Not just pooling, but sharing. Multiple hosts can have a coherent view of the same memory space.
Think about what this does for All-Reduce operations in AI training. Instead of copying data across InfiniBand between GPUs, the GPUs can simply point to a shared CXL memory region where the gradients are being aggregated.
Deep Dive: The CXL-Enabled AI Supercluster Architecture
Let’s map out what a next-gen AI supercluster looks like with CXL integration.
The Hardware Stack
- Compute Tier: NVIDIA Blackwell (B200) or AMD Instinct MI300X nodes. These units handle the heavy matrix multiplication.
- Local Tier: HBM3e on-chip for the “hot” weights and activations.
- CXL Tier (The Expansion): A bank of E3.S CXL memory modules (like those produced by Samsung or SK Hynix) connected via a CXL switch (e.g., Astera Labs or Broadcom).
- The Fabric: A CXL 3.0 leaf-spine architecture connecting all compute nodes to the memory pool.
The Software Logic: Tiered Memory Management
This is where the engineering gets difficult. You can’t just plug in CXL memory and expect magic. The Linux kernel and the AI framework (PyTorch/JAX) need to be CXL-aware.
We use a concept called Tiered Memory. The operating system treats CXL memory as a slower NUMA (Non-Uniform Memory Access) node.
# A conceptual example of how a CXL-aware memory allocator might work
import torch_cxl
# Allocate the "Hot" weights to HBM
model_weights = torch.randn(size, device='cuda:0')
# Allocate the "Cold" KV-Cache or Optimizer States to CXL Memory
# This memory is physically located in a drawer 3 meters away,
# but the GPU sees it as part of its address space.
optimizer_states = torch_cxl.allocate_on_tier(
size,
tier="cxl_memory_pool",
priority="low_latency"
)
# During the backward pass, the CXL controller handles the
# pre-fetching of data into HBM before the GPU needs it.In this architecture, the Fabric Manager is the “brain.” It monitors the memory pressure across the cluster and orchestrates the mapping of CXL memory segments to specific Virtual Machines or Containers.
Why the Hype? (And the Reality Check)
The hype around CXL reached a fever pitch in 2023/2024, with many calling it the “End of the Server as We Know It.” While the potential is massive, the engineering reality is currently in the “Trough of Disillusionment” phase for one primary reason: Hardware Availability.
While CXL 1.1/2.0 is supported on Intel Sapphire Rapids and AMD Genoa/Bergamo CPUs, true CXL 3.0 fabric switches are only just hitting the lab environments.
The “Hype” Context: CXL vs. NVLink
There is a common misconception that CXL is here to kill NVIDIA’s NVLink. This is incorrect.
- NVLink is designed for maximum bandwidth (1.8TB/s on Blackwell) between a small number of GPUs. It is a “scale-up” technology.
- CXL is designed for composable capacity and heterogeneous connectivity. It is a “scale-out” and “disaggregation” technology.
In a premium engineering setup, you use both. You use NVLink for the intense GPU-to-GPU synchronization during a forward/backward pass, and you use CXL to expand the total memory capacity of the cluster so you can fit models that would otherwise require 4x the number of GPUs.
The Engineering Curiosity: The “Missing Link” in LLM Inference
One of the most exciting applications of CXL is in LLM Inference, specifically the KV Cache.
For those who haven’t spent late nights optimizing vLLM or TGI (Text Generation Inference), the KV Cache is the memory used to store the “context” of a conversation. As the conversation gets longer (128k, 1M context windows), the KV Cache explodes. It quickly consumes all available HBM, forcing you to reduce batch sizes, which kills throughput and makes your API expensive.
CXL changes the game for KV Caching:
- Offloading: You can offload the “older” parts of the KV Cache from HBM to CXL-attached DDR5.
- Latency Masking: Because CXL latencies are sub-microsecond, you can pre-fetch the next set of KVs into HBM while the GPU is processing the current token.
- Density: You can support 10x the number of concurrent users on a single GPU node by utilizing a 512GB CXL memory module for the cache, rather than being limited to the 80GB/141GB of HBM.
Technical Hurdle: The Coherency Challenge
If memory pooling sounds too good to be true, it’s because it’s incredibly hard to implement at the hardware level. The biggest challenge is Cache Coherency.
When Node A writes to a shared CXL memory address, how does Node B know that its local cache of that address is now invalid?
- In CXL 2.0, we use a “Home Agent” on the host to manage this.
- In CXL 3.0, we introduce Back-Invalidation. The fabric itself can send messages to hosts telling them to clear their caches for specific addresses.
Implementing this without creating a “broadcast storm” that eats up all your bandwidth is the current frontier of interconnect engineering. It requires sophisticated Fabric Managers that act as the traffic cops for the entire rack.
Building for the CXL Future: What Engineers Should Do Now
If you are building infrastructure for 2025 and beyond, you can’t ignore the CXL roadmap. Here is how the most advanced engineering teams are preparing:
1. Embracing NUMA-Awareness
Software needs to be ready for tiered memory. If your application assumes all “local” memory has the same latency, it will perform poorly on CXL. Tools like numactl and libraries like memkind are becoming essential parts of the AI stack.
2. Standardizing on E3.S Form Factors
The industry is moving away from traditional DIMM slots for expansion and toward EDSFF (Enterprise and Data Center Standard Form Factor) like E3.S. These look like NVMe drives but house CXL controllers and DRAM. They are hot-swappable and offer much better cooling for high-density racks.
3. Rethinking the Rack Power Budget
CXL switches and memory drawers add power overhead. A fully disaggregated rack might require 50kW to 100kW of power. Engineering teams are looking at liquid cooling not just for the GPUs, but for the CXL switching fabric as well.
The Big Picture: Toward the “Data-Centric” Computer
For 40 years, we have built “Computer-Centric” systems: a CPU, some RAM, and some I/O. If you wanted more RAM, you bought a bigger CPU.
CXL is the final nail in the coffin of that architecture. We are moving toward Data-Centric systems where the Memory Fabric is the center of the universe, and CPUs and GPUs are just “processing elements” that plug into it.
In the AI supercluster of 2026:
- Compute is ephemeral: You spin up a GPU, attach it to a 10TB CXL memory slice, run your training job, and release the memory back to the pool.
- Scaling is linear: You don’t buy “servers”; you buy “compute modules” and “memory modules” and plug them into a CXL/Ethernet backplane.
- The “Memory Wall” is broken: We stop fighting the physics of HBM and start leveraging the economics of DDR5 through a high-performance fabric.
The road to CXL 3.0 is paved with complex engineering challenges—signal integrity at PCIe Gen6 speeds, fabric management software, and cache coherency protocols. But the prize is a level of efficiency and scale that makes our current “stranded memory” clusters look like relics of a bygone era.
If you’re an infrastructure engineer, the message is clear: The network is becoming the memory bus. It’s time to start building for the fabric.