The 10 Billion QPS Question: Dissecting Meta's Sharded Load Balancer

You’re scrolling through your feed. A friend posts a photo. You hit ‘like’. In the time it takes for that tiny red heart to appear, a digital tsunami has been unleashed. Your single click is one of ten billion similar operations Meta’s infrastructure must route, process, and resolve every single second. The scale is almost incomprehensible. It’s not just “big data”; it’s a real-time, planet-spanning nervous system where latency is measured in microseconds and global consistency is non-negotiable.

The unsung hero making this possible isn’t a fancy new database or a bleeding-edge AI model. It’s the humble, brutal, and breathtakingly scaled load balancer. But this is no off-the-shelf hardware appliance or a simple Kubernetes Service. This is Meta’s Sharded Load Balancer, a bespoke, globally-distributed routing fabric that forms the absolute bedrock of their services. Today, we’re going to tear down the velvet rope and get an engineer’s-eye view of the machinery that keeps the digital world’s largest party running.

From Monolith to Microservices: The Scaling Crisis That Forged a New Primitive

To understand why this system exists, we need to rewind. In the early days, services were monolithic. A user request would hit a web server, which talked to a single, gargantuan backend. Load balancing was relatively simple: a few hardware boxes (think F5 BIG-IP) distributing traffic across a pool of identical front-end servers.

Then came the microservices explosion. Instagram, WhatsApp, Messenger, core Facebook services—all became distinct entities, each with its own scaling requirements, failure domains, and deployment cycles. The simple “one front-end pool” model shattered. Suddenly, you needed to route traffic not just to servers, but to tens of thousands of individual service endpoints across hundreds of global Points of Presence (PoPs) and massive regional data centers.

The old guard—DNS-based Global Server Load Balancing (GSLB) and traditional Layer 4 (L4) load balancers—buckled under the strain.

• DNS was too slow. TTLs (time-to-live) meant failure detection and rerouting took minutes, not milliseconds.
• Centralized L4 balancers became bottlenecks. They were single points of failure and scaling them vertically hit physical limits.
• Lack of agility. Updating routing rules for a new service deployment across the globe could take hours.

Meta needed a new traffic routing primitive. They needed a system that was:

1. Globally consistent: A user in Delhi should be routed using the same logic as a user in Dallas.
2. Extremely fast: Adds negligible latency (think <1ms).
3. Infinitely scalable: Can grow linearly with traffic.
4. Highly available: Survives data center losses, network partitions, and software bugs.
5. Programmatically agile: Allows engineers to deploy new routing configs globally in seconds.

The answer was a radical re-architecture: sharding the load balancer itself.

Architectural Deep Dive: The Four Pillars of the Sharded Load Balancer

Let’s map the system. Imagine it as a distributed control plane and a hyper-optimized data plane, working in lockstep.

1. The Control Plane: Configerator & Shard Manager

At the heart is the global source of truth: Configerator. This is where engineers define services, pools of backend hosts, and the routing policies that glue them together (e.g., “Route 5% of traffic for service graphql-fe to the new canary pool in prn1”).

# A simplified conceptual config
service: graphql-fe
default_pool: graphql-main-prn1
canary_pool: graphql-nextgen-prn1
routing_policy:
    - rule: header["x-client-version"] == "beta"
      action: route_to(canary_pool)
    - rule: random_sample(5%)
      action: route_to(canary_pool)

Configerator doesn’t push configs. It’s a publisher. The subscriber is the Shard Manager. Its job is to take the global service configuration, understand the current state of the world (which data centers are healthy, which backends are up), and compute the optimal routing table for every single shard in the data plane.

The key innovation: The routing problem is sharded by connection, not by service. The Shard Manager uses a consistent hashing function (like Rendezvous Hashing) on a connection 5-tuple (source IP, source port, dest IP, dest port, protocol). This determines which specific load balancer shard is responsible for that connection’s state and routing decisions. This ensures that all packets for a given connection always land on the same shard, maintaining TCP state consistency without complex synchronization.

2. The Data Plane: The Shard & The Forwarding Engine

This is where the rubber meets the road at 10 billion QPS. Each Shard is a process, typically running on a dedicated server. It has one job: receive packets, consult its locally cached routing table (delivered by the Shard Manager), and forward them at line rate.

But we’re not talking about a Linux user-space process using iptables. This is bare-metal, kernel-bypass performance. Meta heavily leverages DPDK (Data Plane Development Kit) or similar technologies.

• NIC -> User Space: Packets are pulled directly from the NIC into the shard process’s memory, bypassing the kernel network stack entirely.
• Lock-Free Data Structures: The routing table is stored in massive, shared-nothing hash tables and prefix tries, designed for concurrent read access. Lookups must be wait-free.
• CPU Pinning & NUMA Awareness: Shard processes are pinned to specific CPU cores. Memory is allocated on the correct NUMA node nearest the NIC and CPU cores to avoid costly cross-socket memory access. This is where microseconds are won or lost.

Packet Flow in a Shard:
1. Packet arrives on NIC RX queue (bound to CPU Core 5).
2. DPDK poll-mode driver on Core 5 grabs the packet.
3. Core 5 extracts the 5-tuple and performs a consistent hash.
   -> This hash identifies *this shard* as the owner. Proceed.
4. Core 5 does a lookup in the local, read-only Forwarding Information Base (FIB).
   - Destination IP is a Virtual IP (VIP) for service `graphql-fe`.
   - FIB says: "VIP -> Healthy backend at 10.0.5.12:443, weight 100".
5. Core 5 performs Network Address Translation (NAT): rewrites the destination IP/port from the VIP to 10.0.5.12:443.
6. Core 5 places the modified packet on the correct NIC TX queue for egress.

Total added latency: often under 50 microseconds.

3. Health Checking & Failure Detection: The Nervous System

A routing table is useless if it doesn’t know which backends are alive. Meta employs a multi-layer health checking system that is both insanely frequent and surgically precise.

• Proxied Health Checks: Each shard continuously sends lightweight probes (TCP SYN, HTTP/2 PING) to every backend in its table. This is local and fast, but only sees network reachability from that shard’s perspective.
• Centralized Health Service (Pingora-style): A separate, dedicated service performs deeper, application-level health checks (e.g., “can this MySQL host execute a SELECT 1?”). It aggregates this intelligence and feeds it back to the Shard Manager.
• The Magic: Fast Failover. When a shard’s local probe fails, it doesn’t wait for the central service. It can immediately mark the backend as “suspect” and reroute traffic to other hosts in the pool. The central service provides the definitive “this host is dead, remove it globally” verdict. This combination gives sub-second failure detection without causing global routing flaps on transient network blips.

4. Global Traffic Steering: Anycast, ECMP, and The Magic of NetNORAD

How does a packet from your phone in London even find the right shard in a data center in Virginia?

• Anycast BGP: Meta announces the same Virtual IP (VIP) blocks from many of its global PoPs. Your packet gets routed to the topologically nearest PoP. This reduces miles traveled and is great for connection establishment.
• Inside the Data Center: ECMP. Once in a data center, the VIP is not hosted on a single machine. The data center’s network fabric uses Equal-Cost Multi-Path (ECMP) routing. It hashes the packet’s 5-tuple and sprays packets for the VIP across hundreds of shard servers. Remember the consistent hash? The ECMP hash and the shard’s consistent hash are aligned. This ensures the network fabric delivers the packet to the very server whose shard is responsible for that connection. It’s a beautiful dance between network hardware and software.

But what if the nearest PoP is having issues? Enter NetNORAD (Network Notification Of Reachability And Degradation). This is Meta’s global network monitoring brain. It constantly measures latency, loss, and throughput between every PoP and user population centers. If NetNORAD detects that the Paris PoP is experiencing high latency for users in Spain, it can instruct the Shard Manager to adjust weights. Suddenly, traffic from Spain might be steered more heavily to the healthy London PoP, even if it’s slightly farther away. This is application-aware traffic engineering in real-time.

The Numbers: A Symphony of Scale

Let’s put some concrete figures to this architecture to truly appreciate the engineering feat.

• 10 Billion Queries Per Second: This isn’t just HTTP requests. It’s every packet flow, every DNS query, every video chunk request that needs routing. It’s the aggregate throughput across all Meta’s services.
• Millions of Routing Decisions per Second per Shard: A single shard server might handle 1-2 million packets per second. Each requires a stateful lookup and a forwarding decision.
• Sub-Millisecond P99 Latency Added: The entire shard processing pipeline, from packet-in to packet-out, is measured in tens of microseconds. The P99 (99th percentile) latency added is kept under 1 millisecond. In a world where a 100ms delay can reduce user engagement, this is critical.
• Global Configuration Updates in < 1 Second: A new service deployment or a traffic shift rule propagates from Configerator to every shard on the planet in under a second. This is the agility that allows for continuous deployment at a planetary scale.
• Tens of Thousands of Shard Servers: The data plane is comprised of hundreds of thousands of CPU cores, spread across dedicated servers in PoPs and data centers worldwide.

The Curiosities and Trade-Offs: Engineering at the Edge

Building this isn’t just about applying known patterns. It’s about confronting unique, Meta-scale problems.

• The “Thundering Herd” Problem on Config Change: When a popular service’s config changes, every shard in the world recomputes its state simultaneously. If done naively, this could cause a synchronized stampede of health checks to the new backends. The solution involves staggered updates and graceful connection draining, where old routing tables are kept warm for existing connections while new ones use the updated config.
• Stateful Services & Connection Persistence: For stateful protocols (like custom RPC protocols with long-lived connections), the consistent hash is sacrosanct. Losing a shard server means its connections must be gracefully migrated. This involves state handoff between shards or, for truly critical state, relying on backend application-level reconnection logic.
• Hardware vs. Software: Why not just use custom ASICs (like Google’s Jupiter)? The trade-off is flexibility vs. efficiency. A software-based DPDK shard can be updated multiple times a day with new routing features, protocol support, or bug fixes. An ASIC’s logic is frozen in silicon for years. At Meta’s scale and pace of innovation, software-defined networking (SDN) at the host level wins.
• Debugging a Planetary System: How do you debug a misrouted packet when the system spans the globe? The answer is massive, structured logging and tracing. Every shard logs its decisions (at a sampled rate) to a central telemetry system. Tools like Scuba (Meta’s real-time analytics database) allow engineers to query for a specific user’s request flow across PoPs and shards in seconds, reconstructing the entire routing path.

The Ripple Effect: Why This Matters Beyond Meta

The Sharded Load Balancer isn’t just a piece of internal plumbing. It represents a paradigm shift in how we think about cloud-scale infrastructure.

1. The Death of the Centralized Gateway: It proves that the classic “API Gateway” or “Load Balancer” as a centralized cluster is an anti-pattern at extreme scale. The future is sharded, decentralized data planes with a smart control plane.
2. The Primacy of the Control Plane: The real intellectual property is in the Shard Manager and Configerator—the software that can compute and distribute perfect, consistent routing tables globally in real-time. This is the pattern behind modern service meshes (like Istio) but built for a scale orders of magnitude larger.
3. Infrastructure as a Competitive Moat: This system isn’t something you can rent from a cloud provider (yet). It’s a decade of accumulated engineering solving problems that only appear at the very edge of technological possibility. It directly enables features like seamless global failovers, instant rollout of new features, and the consistent, low-latency experience users expect.

The Horizon: What’s Next for the Routing Fabric?

The work is never done. The next frontiers are already in sight:

• QUIC/HTTP3 Ubiquity: These protocols break the traditional 5-tuple connection model. Load balancers must evolve to route based on connection IDs, requiring new state management and hashing strategies.
• eBPF as a Shard Component: Could the ultra-fast packet processing logic of a shard be written as eBPF programs, loaded into the kernel, and managed by the same control plane? This could reduce context switches and push latency even lower.
• AI-Driven Traffic Engineering: What if NetNORAD and the Shard Manager were powered by predictive models? They could pre-emptively shift traffic away from a PoP before a fiber cut happens, based on patterns and external data.

So, the next time you tap ‘like’ and that heart flashes instantly, remember the invisible journey. Your tap triggered a hash function in a NIC in a PoP a hundred miles away, which selected a specific core on a specific server, which consulted a routing table delivered seconds earlier from a global control plane, all to send your affirmation on its way in less time than it takes for a neuron to fire. That’s the magic. Not in the feature, but in the foundation. The Sharded Load Balancer is the silent, hyper-competent stage manager for the entire show, and it’s one of the most impressive pieces of infrastructure software ever built.