The Great Unbundling: Why Your Data Center Should Think Like a Neural Network, Not a Server

Or: How RoCEv2 Turned Memory Into a Pool Party and Compute Into a Hired Gun

You know that sinking feeling when you’re staring at a 128-core server, and you realize 80% of its DRAM is just… sitting there? Idle. Waiting. While some other node across the rack is gasping for memory, paging to a local SSD at the speed of sadness.

For years, we accepted this. We built monoliths. We bought $500,000 servers with 4TB of RAM because one application needed it—and then watched 3.8TB go unused during the nightly batch job. The fundamental law of the data center was: you get the compute you paid for, and the memory that came glued to it.

But the hyperscalers (Facebook, Google, Microsoft, AWS, Alibaba) looked at that waste and said: “No. We are unbundling this marriage.”

Welcome to the world of disaggregated memory and compute. And the unsung hero that makes it possible? A networking protocol originally designed for storage, now riding a 100Gbps wire with latency so low it makes local DRAM nervous. I’m talking, of course, about RDMA over Converged Ethernet v2 (RoCEv2) .

This isn’t some theoretical paper from a 2025 ACM conference. This is live in production at scale. Let’s crack open the chassis.

The Hyperscale Problem: The Memory Wasteland

First, let me paint the problem in brutal numbers.

In a typical hyperscale data center, a server with 256GB of DRAM might run a mix of latency-sensitive services (web serving, caching) and throughput-oriented batch jobs. The memory utilization across the fleet? Often below 50% . Why? Two dirty little secrets:

1. Fragmentation at scale: You provision for peak load. You over-allocate for failure domain isolation. You end up with memory stranded on under-utilized hosts.
2. The “sticky memory” tax: Memory is physically pinned to the compute socket. To use more memory, you buy more servers. To swap a memory-heavy workload? You have to move the entire VM or container, not just the data.

Enter the radical idea: What if memory was a network-attached resource?

You pull the DRAM out of the servers. You put it into a memory blade—a dense, power-efficient box that is just DRAM, a controller, and a network card. Compute servers (the “CPU pools”) connect to this memory pool over a high-speed, low-latency fabric.

Now, a compute node can access 2TB of memory that physically lives three racks away, as if it were local. The compute node itself might only have 32GB of local (L1/L2) scratch pad. This is Disaggregated Memory—often called memory pooling or far memory.

But here’s the kicker: You cannot do this with TCP/IP.

Why TCP/IP Dies Here

TCP is a chunky, kernel-touching, context-switching disaster for this use case. Every read from far memory would require:

1. System call (read()).
2. Kernel buffer copy.
3. TCP stack processing (checksum, congestion control, ACKs).
4. Interrupt delivery.
5. Data copy to user space.

Latency? 10–50 microseconds. For a memory load that should take 100 nanoseconds? That’s a 100x–500x penalty. Unacceptable.

We need a protocol that lets us read a remote memory location in single-digit microseconds—ideally under 3µs—without the kernel getting involved at all. We need Remote Direct Memory Access (RDMA) .

RoCEv2: The Protocol That Ate the Network

Let’s get the nomenclature straight. There are two dominant RDMA transports:

• InfiniBand: The gold standard. Native RDMA. Zero-loss, credit-based flow control. But requires specialized switches and HCAs (Host Channel Adapters). Expensive. Locked into a specific ecosystem.
• RoCEv2: RDMA over Converged Ethernet. It takes the InfiniBand transport (the RDMA verbs) and wraps them in a standard Ethernet and UDP/IP header. RoCEv2 uses a UDP destination port (4791) to identify RDMA traffic.

Why is RoCEv2 winning the data center? Because it runs on commodity Ethernet. You don’t need InfiniBand switches. You can use your existing Mellanox/Chelsio/Broadcom NICs (the high-end ones, obviously) and your standard Arista/Cisco/Juniper leaf-spine fabric, as long as you enable a few critical features.

The magic of RoCEv2 lies in three hardware features:

1. Priority Flow Control (PFC): This is the controversial one. PFC is a Layer 2 mechanism that says: “Hey switch, if my ingress buffer is getting full for the RDMA traffic class, tell the sender to PAUSE for a few microseconds.” It creates a lossless fabric for that class. Without this, a single dropped packet destroys RDMA performance (and often crashes the session).

   • The curse: PFC can cause head-of-line blocking and deadlocks if misconfigured. It requires perfect network engineering. You must monitor for PFC storms (where the pause frame itself causes congestion). Many engineers hate PFC. But it works at scale when tuned well.

2. Explicit Congestion Notification (ECN): Instead of dropping packets or pausing, ECN marks packets as “experiencing congestion.” The receiver sees the mark, and the RDMA sender (via the CC algorithm) slows down. This is reactive, not preventative. It’s used in the DCQCN (Data Center Quantized Congestion Notification) algorithm, which is basically the standard for RoCEv2 congestion control.

3. Hardware Offload (the real hero): The NIC (SmartNIC/DPU) has a RDMA engine on chip. It handles the entire transport layer. The packet arrives, the NIC parses the UDP header, identifies it as RoCEv2, checks the destination QP (Queue Pair), and performs a DMA write directly into the host memory at the address specified in the packet. The CPU is never interrupted. Zero. Bounce. Copies.

Result: With a good NIC (like a Mellanox ConnectX-6 or newer) and a thin fabric, you get 1.5 – 3 µs one-way latency for a memory read. That’s close to local DRAM latency (80–100ns) times 20. But critically, it’s far faster than a local NVMe SSD (10µs+) . It’s in the gap between local DRAM and local Flash. We call this near-coprocessor performance.

The Architecture: A Tale of Two Planes

Let me lay out the actual topology I’ve seen in production at a major social media company (not naming names, but their infrastructure team publishes papers on this).

We have two distinct network planes:

1. The Control Plane (Ethernet/IP – Slow, Smart)

• Purpose: Orchestration, booting, health checks, memory allocation, failure handling.
• Protocol: Standard TCP/IP (HTTP/gRPC).
• Latency: Acceptable (milliseconds).
• Components: A central Memory Scheduler (maybe a distributed service like Facebook’s Memsys or a variant of Kubernetes + a custom scheduler).

2. The Data Plane (RoCEv2 – Fast, Dumb)

• Purpose: Actual memory reads/writes for application payloads.
• Protocol: RoCEv2 (UDP port 4791).
• Latency: 1.5-5µs.
• Components:
  • Compute Nodes: Have a tiny local DRAM cache (L3 cache, basically). They hold a page cache of recently used remote pages.
  • Memory Nodes: Giant boxes of DRAM. 1-4TB per node. They run a lightweight OS that just serves memory pages over RDMA. They expose a memory region (a contiguous virtual address space).
  • The Fabric: A Clos (leaf-spine) topology. All switches must support PFC, ECN, and a lossless RoCE configuration. The Spine is typically 100G or 400G.

How a Memory Request Flows (Step-by-Step)

1. Application on Compute Node issues a malloc() or a mmap() for a large allocation. The library (e.g., a custom libmemkind or Intel’s Optane DCPMM driver adapted for remote RAM) sees the request is too big for local cache.

2. The Library calls the Memory Scheduler (Control Plane) and says: “I need 2GB of memory.” The Scheduler looks up its global memory map, finds a Memory Node with free capacity (perhaps on a different rack in the same Pod), and returns a RKey (Remote Key) and a virtual address on that Memory Node.

   Nerd note: The RKey is an authorization token. It tells the NIC: “This compute node is allowed to read/write this specific memory window.” The NIC will refuse any RDMA request without a valid RKey. This is hardware-level security.

3. The Compute Node’s RDMA NIC now has the RKey and remote address. When the application touches that pointer, the CPU generates a page fault.

4. The Kernel Fault Handler (or a user-space datapath using DPDK/SPDK and userspace RDMA verbs) sees the fault is for a remote page. It constructs an RDMA READ request.

5. The NIC sends a single Ethernet frame (MTU 1500 or 9000 jumbo) with:

   • Destination MAC: The Memory Node’s NIC.
   • Destination IP: The Memory Node’s IP.
   • UDP Port: 4791.
   • RDMA Header: Contains QP number, packet sequence number, RKey, and the target virtual address on the Memory Node.

6. The Memory Node’s NIC receives the packet. Without involving the CPU, it performs a DMA read from its own DRAM at that address, constructs an RDMA READ RESPONSE packet, and sends it back.

7. The Compute Node’s NIC receives the response. Again, no CPU. It writes the data directly into the application’s page (the virtual address the app was trying to access). The page fault is resolved.

Total time: ~2-4 microseconds. The application only sees a slightly higher latency for that one memory access. Subsequent accesses to the same page might be cached locally.

The Strategic Twist: Why This is a Game Changer

This isn’t just about better utilization. This architecture unlocks radical new operational models.

1. The “Infinite” Memory Node

For workloads like Spark, Presto, or even large language model (LLM) inference, the compute pod can now have a virtual memory size that far exceeds physical RAM. The library pages in/out automatically over the RoCEv2 fabric. If you have 10 compute nodes and 2 memory nodes (each with 4TB), your compute pool has 8TB of RAM available. You can run a single Spark job that uses 7TB of distributed memory without ever touching disk.

2. Live Migration in Seconds

Remember the sticky memory tax? If you want to migrate a VM or container, you used to have to copy its entire RAM image (tens of GBs) over the network—taking minutes. With disaggregated memory, the compute node is just the CPU and cache. The memory lives in the pool. To migrate, you just:

• Stop the application.
• Flush the local cache.
• Update the memory scheduler to point the virtual addresses to the new compute node.
• Start the application on the new compute node.

Migration time: < 1 second. The remote memory never moved. This is wild for fault tolerance and load balancing.

3. Memory Oversubscription (The DBA’s Dream)

Database administrators have a theological attachment to having all data in memory. With disaggregation, you can give a MySQL instance a 512GB virtual footprint, but the physical memory pool might only be 300GB. The rest is backed by a fast NVMe local cache on the compute node. The RDMA fabric handles the paging. The DBA doesn’t know the data is remote. They just see 512GB of “memory” with slightly higher tail latency during cache misses.

The Engineering Nightmares (Because It’s Not Perfect)

Let’s be honest. This architecture is beautiful on paper. In the real world, we fight dragons.

The PFC Deadlock Problem

I mentioned PFC. Here’s the horror story: You have a 3-tier Clos fabric. A top-of-rack switch’s buffer fills up for the RoCEv2 class. It sends a PAUSE frame to the leaf. The leaf sends a PAUSE to the spine. The spine sends a PAUSE to another leaf. Now you have a credit loop. No one can send. The entire RoCEv2 fabric stalls. All RDMA traffic stops. Your distributed database starts throwing “Remote Memory Unreachable” errors.

The fix: You must use per-priority PFC and congestion-aware routing (like ECMP + flowlet switching). You also deploy PFC watchdog daemons that monitor pause frames. If a switch pauses for more than X milliseconds, it drops the offending flow (sacrificing one packet to break the deadlock). This is a monumental networking engineering feat to tune correctly. Facebook (Meta) has published entire papers on their PFC-less alternatives (ex: HPCC - High Precision Congestion Control). Many newer fabrics are moving to lossy RDMA (no PFC) and relying on smarter ECN/CC.

The Cache Coherency Headache

This is NOT cache coherent. There is no hardware snooping between compute nodes. If Node A writes to a memory address on the memory node, Node B will not see that write until its local cache is invalidated. This means you cannot use this for classic shared-memory multiprocessing (OpenMP, pthreads sharing a giant variable). You must use distributed programming models (MPI, RPC, atomic operations via RDMA) to manage data ownership.

The clever workaround: Many systems use RDMA atomics (fetch-and-add, compare-and-swap) to build distributed locks and counters. The NIC can perform these operations on the memory node directly, without CPU involvement. It’s a poor man’s atomic memory.

The NIC Memory Footprint

Your RDMA NIC has a small amount of local SRAM for connection state. Each RDMA connection (Queue Pair) consumes roughly 1-2KB of NIC memory. A modern NIC might support 1 million QPs. But if you have 500 compute nodes talking to 100 memory nodes, you might have 50,000 QPs. Manageable. But if you use one QP per thread (like a web server with 10,000 threads), you blow the QP budget. Design your QP model carefully. Usually, one QP per core, or one QP per thread pool, is the pattern.

The Future: CXL vs. RoCEv2 – The Unlikely Alliance

You might have heard of Compute Express Link (CXL) . It’s a cache-coherent interconnect over PCIe and over a fabric. CXL promises memory pooling with hardware cache coherency. It feels like the “natural” evolution.

So why is RoCEv2 still winning in hyperscale clusters right now ?

• CXL Switches are expensive and early. The first generation of CXL memory pooling (Type 3 devices) are just reaching production. They are limited in scale (typically a single rack).
• CXL has a distance limitation. It’s a PCIe-based protocol. Over a copper cable, you might get 5-10 meters. Over a retimer (re-timing driver), maybe 30 meters. RoCEv2? As far as your Ethernet fabric can go (kilometers if you use DWDM optics).

The emerging hybrid architecture (which I think is the real future):

1. Local, Fast, Coherent: Use CXL to pool memory within a rack (e.g., 8 servers sharing 16TB of CXL-attached memory). Latency < 500ns. Perfect for workloads that need cache coherence.
2. Global, Large, Uncoherent: Use RoCEv2 to pool memory across racks, rows, and data halls. Latency < 5µs. Perfect for big data, AI training, and far-memory paging.

This two-tier approach is already being prototyped at companies like Microsoft (their Project Bifrost) and Google (their TPU memory architecture heavily relies on RDMA between pods) .

Why Should You Care?

If you run infrastructure at a scale where 5% utilization improvement saves you $10M a year, this architecture is not optional—it’s survival. But even at smaller scales, the lessons apply:

1. Your network is the new motherboard. Stop treating it as a dumb pipe. Your NIC is a second CPU. Your switch is a memory controller. Think in terms of fabric-attached resources, not servers.
2. Latency is the new capacity. You can throw more hardware at a problem, but you can’t fix a bad fabric. RoCEv2 forces you to build a deterministic, low-jitter network. That benefits everything.
3. Storage is dead, long live memory. The line between RAM and storage is blurring. Disaggregated memory is the first step toward a disaggregated data center where every resource (GPU, FPGA, CPU, memory, storage) is a pool on a wire. The server is just an abstraction.

The Final Take

RoCEv2 isn’t sexy. It’s not a shiny new protocol. It’s an old dog (InfiniBand) wearing a new Ethernet hat. But it’s the workhorse that makes the unbundled data center possible. It turns your network into an extension of your memory bus.

When you next look at a server and see those DIMM slots, realize: they are now optional. The real memory is out there, on the network, waiting for a compute node to ring its doorbell with a low-latency UDP packet.

And the best part? You can build this today. Get a ConnectX-6, a pair of Arista 7280 switches, and a few servers with large DRAM. Enable PFC and ECN. Write some code using librdmacm. You’ll be paging over the network at 100 Gbps before lunch.

Just be ready for the PFC deadlock nightmares.

Now go forth and disaggregate.