In the world of high-frequency trading and real-time recommendation engines, microseconds aren’t just a metric—they are the margin between a market-leading product and a legacy system. As machine learning models transition from static batch processing to hyper-dynamic, real-time inference, the “Feature Store” has become the heart of the modern AI stack.
But here is the dirty secret of petabyte-scale feature stores: Your network is fast, your disks are fast, but your CPU is drowning in busywork.
When you are pulling feature vectors for a ranking model at a scale of 10 million requests per second, the traditional Linux networking stack becomes your greatest enemy. Every time a packet travels from the Network Interface Card (NIC), through the kernel, and into your application’s user space, you pay a “latency tax” in the form of context switches and memory copies (memcpy).
At petabyte scale, these copies consume more CPU cycles than the actual feature engineering itself. To solve this, we had to rethink the data plane entirely. This is the story of how we implemented a zero-copy architecture using RDMA (Remote Direct Memory Access) and eBPF (Extended Berkeley Packet Filter) to move data at the speed of hardware.
The Ghost in the Machine: Why TCP/IP Fails at Scale
In a standard architecture, fetching a feature looks like this:
- The application requests a feature vector.
- The kernel receives the request via a system call.
- The data is read from the disk/cache into a kernel buffer.
- The kernel copies the data from the kernel buffer to the user-space application buffer.
- The application processes the data.
When your features are small, this is fine. When your features are massive embeddings (vectors with thousands of dimensions) and you have billions of them, the CPU spends 40% of its time simply moving bits from one memory address to another. This is the Memory Wall.
Furthermore, the standard TCP/IP stack is heavy. It handles congestion control, retransmission, and packet ordering in software. At 100Gbps or 200Gbps line speeds, a single CPU core cannot keep up with the interrupt processing required to handle that volume of packets.
We needed a way to bypass the kernel entirely.
Enter RDMA: The “Skip the Middleman” Strategy
RDMA allows one computer to read or write directly into the memory of another computer without involving either one’s operating system. There is no context switching, no kernel intervention, and—most importantly—zero copies.
The RoCE v2 Breakthrough
We settled on RoCE v2 (RDMA over Converged Ethernet). Unlike InfiniBand, which requires specialized (and expensive) networking hardware, RoCE v2 runs over standard Ethernet. It wraps RDMA transport headers inside UDP/IP packets, allowing it to be routed across modern data center fabrics.
In our petabyte-scale feature store, we utilize Direct Data Placement (DDP). When a compute node needs a feature vector from a storage node, the NIC on the storage node reads the data directly from the RAM and places it directly into the RAM of the compute node.
The CPU on the storage node literally never sees the data. It just gets a notification (a “Completion Queue Entry”) that the transfer is done.
Registering the Memory Region (MR)
The magic of RDMA starts with Memory Registration. To allow the NIC to access memory, we must “pin” the memory pages so the OS doesn’t swap them to disk.
// High-level conceptual example of RDMA memory registration
struct ibv_mr *mr;
size_t size = 1024 * 1024 * 1024; // 1GB Feature Buffer
void *buffer = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
// Register memory with the RDMA device
mr = ibv_reg_mr(pd, buffer, size, IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ);
if (!mr) {
fprintf(stderr, "Error: Could not register Memory Region\n");
return 1;
}By registering these large chunks of memory, we create a “Feature Pool.” The hardware handles the address translation, ensuring that the remote node can only access the specific memory offsets we permit.
The eBPF Factor: Programmable Data Planes
If RDMA is the high-speed highway, eBPF is the intelligent traffic controller. RDMA is great for bulk data transfer, but how do we handle the “control plane”—the logic that decides which node has which feature?
Traditional load balancers or service meshes (like Envoy or Istio) add milliseconds of latency. For a feature store, that’s unacceptable. We integrated eBPF at the XDP (Express Data Path) level.
XDP allows us to run eBPF code directly on the NIC’s driver, before the packet even reaches the Linux kernel.
Why eBPF for Feature Routing?
When a client asks for a feature, our eBPF program inspects the incoming packet. If the packet is a “Feature Lookup” request, the eBPF program:
- Hashes the feature ID.
- Looks up the metadata in a BPF Map (shared memory between kernel and user space).
- Directly redirects the packet to the specific storage node that holds that shard of the data.
This happens in nanoseconds.
// A simplified XDP snippet for routing feature requests
SEC("xdp_feature_router")
int xdp_router_func(struct xdp_md *ctx) {
void *data_end = (void *)(long)ctx->data_end;
void *data = (void *)(long)ctx->data;
struct ethhdr *eth = data;
if (eth + 1 > data_end) return XDP_PASS;
// Extract custom header containing Feature ID
struct feature_req_hdr *f_hdr = data + sizeof(struct ethhdr) + sizeof(struct iphdr) + sizeof(struct udphdr);
if (f_hdr + 1 > data_end) return XDP_PASS;
// Fast lookup in BPF Map to find the target RDMA-enabled node
__u32 *node_ip = bpf_map_lookup_elem(&feature_shards_map, &f_hdr->feature_id);
if (node_ip) {
// Rewrite destination IP and redirect back to NIC
rewrite_dst_ip(eth, *node_ip);
return XDP_TX;
}
return XDP_PASS;
}By combining eBPF for routing and RDMA for transfer, we created a Kernel-Bypass End-to-End Pipeline.
Infrastructure at Scale: The “Petabyte” Problem
Building a zero-copy store for 10GB is easy. Doing it for a Petabyte requires solving three massive engineering hurdles: Memory Fragmentation, Congestion Control, and Cache Locality.
1. Solving the Hugepages Crisis
RDMA requires pinned memory. If we use standard 4KB pages, the Translation Lookaside Buffer (TLB) on the CPU becomes a massive bottleneck. For a petabyte store, the TLB miss rate would skyrocket.
We moved to 1GB Hugepages. By using 1GB pages, we reduce the number of entries the hardware needs to track by a factor of 250,000. This ensures that the NIC’s address translation cache is always hitting, maintaining that sub-10 microsecond latency.
2. Taming the Network: DCQCN and PFC
RDMA is “lossless.” Standard Ethernet is “lossy” (it drops packets when congested). If a packet is dropped in an RDMA stream, the performance collapses as the hardware waits for retransmission.
To prevent this at scale, we implemented Priority Flow Control (PFC) and Data Center Quantized Congestion Notification (DCQCN).
- PFC acts like a “pause button” on the switch. If a buffer is full, the switch tells the sender to stop for a few microseconds.
- DCQCN uses eBPF-monitored metrics to tell the RDMA NICs to slow down before the switch buffer overflows.
3. NUMA-Awareness: The Silent Killer
In a dual-socket server, if your NIC is connected to CPU 0, but your feature data is in memory controlled by CPU 1, the data must cross the QPI/UPI link between processors. This adds ~100ns of latency and creates a massive bottleneck.
Our feature store is NUMA-aware. Our eBPF load balancer understands the topology of the storage nodes and ensures that an RDMA transfer only happens between a NIC and the memory directly attached to its local PCIe lane.
The Technical Substance Behind the Hype
There has been a lot of “Zero-Copy” hype lately, especially with the rise of AI-native infrastructure. You might hear vendors claim “O(1) data access” or “infinite bandwidth.”
The reality is that zero-copy is not a magic switch you turn on; it is a fundamental shift in how you manage the lifecycle of a byte. Most systems claim to be zero-copy but still perform a “user-to-user” copy if the data format (like Protobuf or JSON) needs to be deserialized.
To achieve true zero-copy, we adopted Apache Arrow as our in-memory format. Arrow uses a columnar memory layout that is identical on disk, in transit, and in the GPU’s VRAM.
- The storage node reads Arrow buffers from NVMe.
- RDMA transfers the Arrow buffer to the compute node.
- The ML model (using PyTorch or TensorFlow) maps that memory address directly.
Zero serialization. Zero deserialization. Zero copies.
Performance Results: The Proof is in the P99s
When we switched from a standard gRPC/TCP stack to the RDMA + eBPF architecture, the results were transformative:
- Throughput: Increased from 1.2 GB/s per node to 11.5 GB/s (saturating the 100GbE NIC).
- CPU Utilization: Dropped from 85% to 12% for the same workload. The CPU was now free to do what it’s best at: running inference, not moving packets.
- P99 Latency: Reduced from 15ms to 450 microseconds.
The most surprising result wasn’t just the speed—it was the predictability. Standard networking stacks have “jitter” caused by garbage collection or kernel interrupts. RDMA and eBPF are deterministic. Your P99 latency looks almost identical to your P50.
The Engineering Curiosity: Why Now?
Why didn’t we do this five years ago? Two reasons: Tooling and Hardware.
Until recently, writing eBPF required deep C knowledge and a lot of prayer that you wouldn’t crash the kernel. With the arrival of libraries like aya (Rust) and better CO-RE (Compile Once – Run Everywhere) support, eBPF is finally “production-ready” for the average senior engineer.
On the hardware side, the rise of DPUs (Data Processing Units) like the NVIDIA BlueField or AMD Pensando has moved the RDMA and eBPF logic off the host CPU entirely. We are moving toward a future where the “Network Card” is actually a mini-server that handles the entire feature store logic, leaving the main CPU to act purely as an orchestrator.
The Road Ahead
Building a petabyte-scale feature store using RDMA and eBPF is a journey into the “Bare Metal” of computer science. It forces you to abandon the comforts of high-level abstractions and confront the reality of PCIe lanes, memory alignment, and interrupt requests.
As we look to the future, the next frontier is CXL (Compute Express Link). CXL will allow us to pool memory across hundreds of servers, potentially making “Remote” memory feel as fast as “Local” memory.
But for today, if you want to scale to petabytes without burning a hole in your cloud budget or your CPU, you need to stop copying data. You need to let the hardware do its job.
The era of software-defined networking was about flexibility. The era of Hardware-Accelerated Data Planes is about raw, unadulterated performance. And in the race for real-time AI, performance isn’t just a feature—it’s the only thing that matters.