In the world of high-scale AI infrastructure, we’ve spent the last decade perfecting the art of “moving data to compute.” We’ve built massive InfiniBand fabrics, optimized RDMA (Remote Direct Memory Access) kernels, and shaved microseconds off our gRPC stacks. But as we transition from the era of “small” BERT models to the epoch of trillion-parameter Mixture-of-Experts (MoE) and infinite-context LLMs, we are hitting a physical wall.
The wall isn’t just about FLOPs—it’s about the Interconnect Tax.
Right now, if a GPU needs a piece of data from another node’s memory, it has to pack it into a packet, send it across a network stack, unpack it, and store it. This “message-passing” paradigm is the ghost in the machine, causing the tail-latency spikes that ruin real-time AI experiences.
Enter CXL (Compute Express Link) and the shift toward memory-semantic networking. We are moving away from treating the network as a post office and toward treating the entire data center as a single, contiguous pool of addressable memory.
If you’re an infrastructure engineer tasked with serving 100k+ tokens per second at sub-100ms latencies, this isn’t just a hardware upgrade. It is a fundamental rewrite of how we architect AI clusters.
The Dirty Secret of Modern AI: Memory Stranding and the “IO Tax”
Before we dive into the CXL spec, let’s look at the problem. In a typical H100 or B200 cluster, we have what’s called Memory Stranding.
Imagine you have a node with 8 GPUs, each with 80GB of HBM3. You’re running a massive LLM. One GPU might be completely tapped out on memory because it’s holding a massive KV cache (Key-Value cache) for a long-context prompt, while a neighboring GPU is sitting on 20GB of idle memory. In the current PCIe/Ethernet paradigm, that idle 20GB is “stranded.” You can’t easily borrow it without massive performance penalties.
Furthermore, when we scale inference across nodes, we rely on Copy-based semantics. To move a tensor:
- GPU A copies data to System RAM.
- The CPU triggers a NIC transfer.
- Data travels over Ethernet/IB.
- The receiving NIC writes to System RAM.
- GPU B copies from RAM to its local HBM.
Every one of those “hops” introduces serialization delay and jitter. Memory-semantic fabrics seek to eliminate the “copy” entirely. They allow a processor to perform a simple load or store instruction to a memory address that physically resides on a different board, or even a different rack.
CXL: The Protocol That Actually Matters
CXL isn’t a replacement for the network; it’s an evolution of the PCIe physical layer. It runs on the same pins as PCIe Gen5/Gen6 but introduces three distinct protocols:
- CXL.io: The foundational discovery and configuration protocol (basically PCIe with some extras).
- CXL.cache: Allows a peripheral (like a GPU or FPGA) to cache data from the host CPU’s memory with full coherency.
- CXL.mem: This is the “Holy Grail.” It allows the CPU or GPU to access external memory pools (DRAM or Flash) using standard Load/Store semantics.
When we talk about “Memory-Semantic Networking,” we are talking about CXL 3.0/3.1 fabrics. In CXL 3.0, we move from simple point-to-point connections to CXL Switching. We can now build a leaf-and-spine fabric where hundreds of GPUs and thousands of Memory Expansion Modules are interconnected.
The Architectural Deep Dive: AI Serving at Scale
How does this actually change the way we build an inference engine? Let’s look at the three biggest architectural shifts.
1. The Disaggregated KV Cache
In LLM inference, the KV Cache is the primary memory consumer. As the sequence length grows, the memory required to store the attention keys and values grows linearly. This is why your 70B model suddenly runs out of OOM (Out of Memory) when the conversation gets long.
In a traditional architecture, the KV cache must live in the GPU’s HBM. If the HBM is full, you have to swap to system RAM (slow) or kill the request.
With a CXL Memory Fabric, we can build a Global KV Cache Pool.
- The Design: A central rack filled with nothing but E3.S CXL memory modules (terabytes of DDR5).
- The Flow: When a GPU processes a token, it writes the KV pair directly to the CXL fabric.
- The Benefit: Since the fabric supports
load/store, the latency is in the realm of ~200-500 nanoseconds, compared to tens or hundreds of microseconds for a networked swap. This allows for “infinite” context windows where the GPU only holds the active calculation in HBM, while the history lives in the CXL pool.
2. Mixture-of-Experts (MoE) and Zero-Copy Routing
Models like Mixtral or GPT-4 (reportedly) use a Mixture-of-Experts architecture. Only a fraction of the model’s weights are active for any given token.
The challenge? The “Router” decides which “Expert” to send the data to. If the Experts are spread across 8 different nodes, you’re constantly shuffling data over the network.
In a CXL-enabled fabric, all Experts can live in a shared memory space. Instead of “sending” the hidden states to Expert #7 on Node B, Node A simply writes a pointer to the memory address of the hidden state into a shared queue. Expert #7 then performs a load from that address.
No packets were harmed in the making of this inference. We’ve replaced a complex networking stack with a memory controller operation.
3. Unified Coherency and the “Giant GPU”
The ultimate dream of CXL is to make a 128-GPU cluster look like a single machine with 100TB of RAM.
Currently, we use NCCL (NVIDIA Collective Communications Library) to synchronize GPUs. NCCL is brilliant, but it’s still fundamentally bound by the fact that GPU A doesn’t know what’s in GPU B’s memory without an explicit “send/receive” or “all-reduce” operation.
CXL 3.0 introduces Hardware Coherency across the fabric. If one node updates a weight or a bias in the shared pool, the hardware-level “snoop” protocols ensure that every other node’s cache is invalidated or updated.
Let’s Look at the Code: Programming the Fabric
What does this look like for a systems engineer? You won’t be writing standard socket.send() calls. Instead, you’ll be interacting with memory-mapped regions.
Imagine a hypothetical C++ allocator designed for a CXL fabric:
// Traditional approach: Allocate on GPU HBM
void* gpu_ptr;
cudaMalloc(&gpu_ptr, 1024 * 1024 * 100); // 100MB
// CXL Fabric approach: Allocate in the global memory pool
// This memory is physically located on a CXL memory expander node
cxl_mem_handle_t handle = cxl_fabric_alloc(GLOBAL_POOL_ID, 1024 * 1024 * 1000); // 1GB
// Map the fabric memory into the local process virtual address space
void* shared_cache = cxl_mmap(handle);
// Now, we can treat this like local RAM
// The hardware handles the PCIe/CXL protocol translation
memcpy(shared_cache, local_tensor_data, 1024);
// On a DIFFERENT node, we can access the same data
// No network stack required!
void* remote_view = cxl_mmap(handle);
float first_weight = ((float*)remote_view)[0];The magic here is that first_weight = ((float*)remote_view)[0] results in a hardware-level Memory Read request across the CXL fabric. There is no OS kernel involvement, no TCP/IP overhead, and no context switching.
Hype vs. Reality: Why isn’t this everywhere yet?
If you follow tech news, CXL is the “it” word. But there’s a gap between the hype and the production floor.
The “Latency Floor”
While CXL is much faster than Ethernet, it’s still slower than local HBM3. HBM3 on a GPU has bandwidth in the terabytes per second. CXL over PCIe Gen5 is roughly 32GB/s to 64GB/s per x16 link. Substance: CXL isn’t meant to replace HBM; it’s meant to replace the “Swap to Disk” or “Send over Network” paths. It’s a new tier in the memory hierarchy:
- L1/L2/L3 Cache (Fastest)
- HBM3 (High Bandwidth)
- Local DDR5 (High Capacity)
- CXL Fabric Memory (Huge Capacity, Shared)
- SSD/NVMe (Storage)
The Switch Problem
To build a true fabric, we need CXL switches. While companies like Astera Labs, Broadcom, and Marvell are sampling silicon, we are only just seeing the first generation of high-port-count CXL switches. Without the switch, CXL is just a fancy point-to-point cable.
Software Support
The Linux kernel has had CXL support for a few versions now, but the User-space libraries (like a CXL-aware PyTorch or a memory-semantic NCCL) are still in their infancy. We are currently in the “Early Adopter” phase where infrastructure giants (Hyperscalers) are writing their own proprietary drivers to manage these pools.
The Engineering Curiosity: The “Flit” and the Physical Layer
One of the most fascinating technical details of CXL is how it achieves low latency. PCIe is traditionally a “lossless” protocol but has significant overhead due to its legacy.
CXL uses FLITs (Flow Control Units) of 256 bytes. In CXL 3.0, these flits are processed with Low-Latency Forward Error Correction (FEC). This is a critical engineering trade-off:
- By using fixed-size flits, the hardware can begin processing the header of the next packet before the current one has even finished arriving.
- This “cut-through” switching at the memory level is what allows us to keep the added latency of the fabric to under 50ns per switch hop.
When you add that to the DRAM access time, you’re looking at a total remote memory access time of ~250ns. For context, a typical round-trip over a highly optimized 100G Ethernet network is ~5,000ns to 10,000ns. That’s a 20x to 40x improvement.
How This Redefines “Scaling”
When we talk about “Scaling” AI today, we usually mean adding more nodes and doing Data Parallelism or Model Parallelism.
In the CXL era, we will talk about Composable Infrastructure.
Instead of buying a “Server” with a fixed ratio of CPU/GPU/RAM, you will have:
- Compute Racks (Pure GPU/TPU nodes).
- Memory Racks (Pools of CXL-attached DDR5/LPDDR5).
- Fabric Racks (The CXL switching backplane).
If a specific model serving task is “Memory Bound” (like long-context LLMs), the orchestrator (Kubernetes with a CXL-plugin) will dynamically map more “Memory LUNs” from the Memory Rack to the Compute Rack.
This solves the “Stranded Power” problem. We no longer need to over-provision every node with 2TB of RAM “just in case.” We provision the average and burst into the fabric.
The Infrastructure Manager’s Playbook
If you are designing the next generation of an AI serving platform, here is how you should be thinking about the transition to memory-semantic fabrics:
- Move away from monolithic nodes: Start looking at chassis that support CXL 2.0+ E3.S backplanes. Even if you don’t have the switches yet, having the physical slots for memory expansion is key.
- Audit your Tail Latency: 99th percentile (P99) latency in AI serving is usually caused by “The Stall”—waiting for a weights-swap or a KV cache fetch. Map these stalls to see if they could be solved by a 300ns memory load vs. a 10ms disk/network load.
- Prepare for “Heterogeneous Memory”: Your software needs to be “NUMA-aware on steroids.” You’ll have local HBM (Fastest), local RAM (Fast), and Fabric RAM (Slower but massive). Your allocators need to be smart enough to put the “hot” tensors in HBM and the “warm” KV cache in CXL.
A New Era of Distributed Systems
The move to CXL and memory-semantic networking represents a shift in the very physics of distributed computing. We are finally breaking the “Von Neumann Bottleneck” at the data center scale.
For decades, the CPU/GPU was at the center, and memory was just a local peripheral. In the new architectural paradigm, Memory is the Fabric, and compute nodes are just “clients” that attach to it.
For AI model serving, this is the difference between a system that “chokes” on long-context queries and one that scales effortlessly. We are moving from a world of “Message Passing” to a world of “Universal Memory.” And if you’re building in this space, it’s time to stop thinking in packets and start thinking in addresses.
The network is no longer a pipe. The network is now a memory controller. Welcome to the fabric.