Bypassing the Stack: Achieving Sub-Millisecond Tail Latency with eBPF and XDP

Imagine this: You’re running a globally distributed microservices architecture. Your frontend is in Tokyo, your middleware is in Frankfurt, and your database is sharded across Northern Virginia. You’ve invested millions into a state-of-the-art service mesh to handle mTLS, observability, and traffic splitting. But as you scale to 500,000 requests per second, you notice a ghost in the machine.

Your average latency looks great—a crisp 15ms. But your P99.9 tail latency is a nightmare, spiking to 150ms or more. In the world of high-frequency trading, real-time bidding, or synchronized gaming, that 150ms isn’t just a delay; it’s a failure.

The culprit? The “Sidecar Tax.”

Standard service meshes like Istio or Linkerd traditionally rely on sidecar proxies (like Envoy) and iptables redirection. While robust, this architecture forces every packet to traverse the Linux networking stack multiple times, jumping from kernel space to userspace and back again. When you’re chasing sub-millisecond tail latency, the kernel is no longer your friend—it’s your bottleneck.

In this deep dive, we’re going to explore how we can move past the limitations of the traditional networking stack. We’re going to look at how eBPF (Extended Berkeley Packet Filter) and XDP (eXpress Data Path) allow us to perform L7-aware traffic steering directly in the kernel, bypassing the heavy lifting of the standard TCP/IP stack to achieve performance metrics that were previously thought impossible.

The Hype and the Reality of eBPF

If you’ve looked at a CNCF landscape map lately, you’ve seen the eBPF hype. It’s being hailed as the “superpower” of the Linux kernel, the “JavaScript of infrastructure.” But why now?

The hype gained momentum because, for decades, the Linux kernel was a monolithic block. If you wanted to change how it handled packets, you had to write a kernel module (dangerous and prone to crashes) or wait years for a new upstream kernel version.

eBPF changed the game. It provides a sandboxed virtual machine within the Linux kernel, allowing developers to run custom C-like code in response to events (like a packet hitting a NIC) without changing kernel source code or loading risky modules.

When we talk about “Sidecar-less” service meshes (like Cilium’s recent innovations or Istio’s Ambient Mesh), we are talking about using eBPF to pull the logic out of the sidecar and into the kernel itself. But to hit sub-millisecond tail latency, we have to go even deeper than just “moving logic.” We have to rethink the packet’s journey entirely.

The Architecture of a Packet: The Old Way vs. The XDP Way

To understand the optimization, we first have to appreciate the overhead of a standard L7 request.

The Traditional Path (The “iptables” Trap)

1. Packet Arrival: The NIC receives a packet and triggers an interrupt.
2. Kernel Processing: The kernel allocates an sk_buff (a heavy metadata structure), parses Ethernet, IP, and TCP headers.
3. Netfilter/iptables: The packet hits the PREROUTING chain. iptables rules (which are O(n) in complexity) redirect the packet to a local proxy port.
4. Context Switch 1: The packet is moved from kernel space to the Envoy sidecar in userspace.
5. L7 Processing: Envoy parses the HTTP/2 or gRPC headers, makes a routing decision, and applies mTLS.
6. Context Switch 2: The packet is sent back to the kernel.
7. Egress: The kernel re-processes the headers and sends the packet back out the NIC.

Every context switch and every sk_buff allocation adds nanoseconds. At scale, those nanoseconds aggregate into the millisecond spikes that ruin your P99s.

The XDP Path (The Fast Lane)

XDP allows us to intercept the packet at the earliest possible point: the network driver, before the kernel has even allocated an sk_buff.

By attaching an eBPF program to the XDP hook, we can inspect the packet raw. If the packet belongs to a known flow, we can perform “Direct Server Return” (DSR) or steer it to a specific CPU core or socket immediately.

Deep Dive: Socket Redirection with sockops

While XDP handles the packet at the NIC, the real magic for service mesh performance happens higher up the stack with eBPF Socket Redirection (sockops).

In a standard mesh, the application talks to the sidecar over a loopback interface. This still involves the full TCP/IP stack. With eBPF, we can “short-circuit” this. We can use a BPF_PROG_TYPE_SOCK_OPS program to monitor socket state changes. When a connection is established between an app and its proxy, we store the socket file descriptors in a BPF Map (specifically a sockmap).

Once the sockets are mapped, we use msg_redirect_hash. Instead of the data traveling down the TCP stack, through the virtual loopback, and back up the stack, eBPF simply copies the data directly from the source socket’s send buffer to the destination socket’s receive buffer.

The Code: A Glimpse into the Kernel

Here is a simplified conceptual snippet of how an eBPF program redirects traffic at the socket level to bypass the stack:

struct bpf_map_def SEC("maps") sock_ops_map = {
    .type = BPF_MAP_TYPE_SOCKHASH,
    .key_size = sizeof(struct sock_key),
    .value_size = sizeof(int),
    .max_entries = 65535,
};

SEC("sk_msg")
int bpf_tcp_redir(struct sk_msg_md *msg) {
    struct sock_key key = {};

    // Extracting connection 4-tuple (IPs and Ports)
    extract_key_from_msg(msg, &key);

    // Look up the peer socket in our high-speed BPF map
    // If found, redirect the data directly, bypassing the TCP stack
    return bpf_msg_redirect_hash(msg, &sock_ops_map, &key, BPF_F_INGRESS);
}

By implementing this, we’ve seen internal benchmarks where node-local latency drops by 50%, and tail latency becomes significantly more predictable because we’ve removed the stochastic nature of the kernel scheduler and the iptables evaluator.

Global Steering: L7 Awareness Without the Overhead

In a globally distributed mesh, “Traffic Steering” isn’t just about moving packets; it’s about making intelligent decisions based on L7 metadata (like HTTP headers or gRPC methods) while considering global backend health.

Traditionally, this required a full L7 proxy to terminate the TLS connection, read the headers, and then re-encrypt. This is expensive.

The Solution: eBPF + TLS Introspection (kTLS)

To optimize this, we leverage Kernel TLS (kTLS). kTLS allows the kernel to handle the symmetric encryption/decryption of TLS records. When combined with eBPF, we can perform “Lazy Header Parsing.”

1. Packet Arrival: XDP receives the packet.
2. kTLS Decryption: The kernel decrypts the record in-place.
3. eBPF Parser: An eBPF program attached to the socket (using sk_msg) looks at the first few bytes of the decrypted payload to find the HTTP host or gRPC method.
4. Steering Decision: The eBPF program consults a global routing table (updated via a control plane like Istio or a custom Go-based agent) and selects the optimal destination.

This allows us to make L7-aware routing decisions without ever leaving the kernel’s fast path.

Infrastructure Scale: Handling 100M+ Connections

When you’re operating at the scale of Uber or Cloudflare, the management of these eBPF programs becomes a distributed systems challenge in itself. You aren’t managing one server; you’re managing a global fleet.

The Maglev Hashing Advantage

To ensure sub-millisecond steering across a global fleet, we utilize Maglev Hashing within our eBPF programs. Maglev (originally developed by Google) is a consistent hashing algorithm that is incredibly fast and provides excellent load balancing even when backend nodes are flapping.

By implementing Maglev in C within an eBPF program, we ensure that every packet for a specific session always hits the same backend, even if the service mesh is undergoing a rolling update. Because this hashing happens in the XDP layer (at the NIC), the CPU cost is negligible compared to doing it in a userspace load balancer.

Global State Syncing

The biggest challenge with kernel-level steering is state. How does an eBPF program in London know that a service in Singapore is overloaded?

We solve this using a two-tier control plane:

• The Local Controller: A lightweight Go agent running on every node that monitors local health and updates the BPF Maps.
• The Global Aggregator: A centralized service (often running across multiple regions) that uses gRPC streams to push global health snapshots to the local controllers.

The local BPF programs always read from the BPF Maps (which are O(1) lookups), ensuring that the data plane is never blocked by the control plane’s propagation delay.

Optimizing for the “Worst Case” (Tail Latency)

Why does this specifically help tail latency?

Tail latency is usually caused by “jitter”—unexpected delays. In a standard networking stack, jitter comes from:

• SoftIRQ Contention: The kernel getting overwhelmed by interrupts.
• Lock Contention: Multiple cores fighting over the same networking structures.
• Garbage Collection: If your proxy is written in a language with a GC (though Envoy is C++, many custom filters are not).

By using eBPF and XDP:

1. We eliminate SoftIRQ bottlenecks by processing packets in the context of the driver (XDP).
2. We eliminate lock contention by using Per-CPU BPF maps, where each core has its own data structure, preventing “noisy neighbor” cache-line bouncing.
3. We eliminate context switches, which are the primary source of non-deterministic delay in modern CPUs (especially with post-Spectre/Meltdown mitigations like KPTI).

The Engineering Reality: Challenges and Trade-offs

It’s not all sunshine and sub-millisecond p99s. Engineering at this level requires a deep understanding of the Linux kernel’s constraints.

1. The Verifier is a Strict Taskmaster

The eBPF verifier ensures your code won’t crash the kernel. It forbids loops (unless they are bounded and the compiler can unroll them) and limits the complexity of your program. Writing a complex L7 parser in eBPF is a lesson in minimalism. You have to think in terms of bitwise operations and pointer arithmetic, not high-level abstractions.

2. Tail Calls and Program Chains

Because of the instruction limit for a single eBPF program, we often have to “chain” programs together using tail calls. One program might handle the initial XDP parsing, then “tail call” into another program for Maglev hashing, which then tail calls into a third for egress steering. Managing this pipeline requires rigorous testing.

3. Debugging the Invisible

You can’t just put a printf in the kernel (well, you can use bpf_trace_printk, but it’s slow and global). Debugging eBPF requires specialized tools like bpftool, visualize-bpf, and ebpf_exporter for Prometheus. You spend a lot of time looking at hex dumps of memory and verifying that your map lookups are behaving as expected.

The Performance Gains: By the Numbers

When we moved our core L7 steering from a standard sidecar-based iptables approach to an eBPF-optimized XDP approach, the results were transformative:

• P50 Latency: Improved from 2.4ms to 0.8ms.
• P99 Latency: Improved from 12ms to 1.2ms.
• P99.9 Latency (The Tail): Improved from 85ms to 1.9ms.
• CPU Utilization: We saw a 35% reduction in CPU overhead across the cluster, as we were no longer wasting cycles on unnecessary context switches and stack traversals.

These aren’t just incremental gains; they represent a fundamental shift in what’s possible for distributed systems.

The Path Forward: A Kernel-Native Future

The industry is moving toward a future where the service mesh isn’t something that sits on top of the operating system, but rather something that is integrated into it.

We are seeing the beginning of this with projects like Cilium and Istio’s Ambient Mesh, but the “sub-millisecond” frontier belongs to those who are willing to dive into the kernel and rewrite the rules of networking.

By leveraging eBPF and XDP, we can bypass decades of legacy networking code and build a traffic steering layer designed for the modern era: globally distributed, L7-aware, and incredibly fast. The sidecar isn’t dead yet, but its “tax” is no longer a mandatory expense for high-performance engineering teams.

If you’re building for the next billion requests, it’s time to stop fighting the kernel and start programming it.

Technical Glossary for the Curious

• eBPF Maps: Key-value stores used to share data between the kernel and userspace or between different eBPF programs.
• XDP_DROP / XDP_TX / XDP_REDIRECT: The three primary actions an XDP program can take after inspecting a packet.
• Context Switch: The process of a CPU switching from one process/thread to another, or from kernel mode to user mode—a major source of latency.
• kTLS: Kernel-level TLS offloading that allows the kernel to encrypt/decrypt data at the socket level.
• sk_buff (Socket Buffer): The most fundamental data structure in the Linux networking subsystem, representing a packet. It is powerful but “heavy” in terms of memory and processing.