The Internet's Biggest Sleight of Hand: Inside Netflix's Open Connect CDN

You hit play on Stranger Things. Within milliseconds, the first frame splashes across your screen. You might think you just requested a video from “the cloud,” but you’d be wrong. You didn’t hit Netflix’s servers; you hit a server sitting in your ISP’s local data center, likely less than 50 miles from your couch.

We tend to think of Netflix as a streaming service, but that’s just the tip of the iceberg. Underneath that interface lies one of the most sophisticated, massive, and aggressive content delivery networks (CDNs) on the planet: Open Connect.

While big tech rivals like Google and Meta were building out massive regional data centers, Netflix took a radically different approach. They didn’t just build a CDN to serve video; they built a logistics network for petabytes of data that effectively turns your internet service provider (ISP) into a Netflix data center.

But how does this actually work? How do they manage to stream 4K HDR content to 260+ million subscribers simultaneously without melting the internet? Let’s pull back the curtain on the architecture, the hardware, and the sheer engineering audacity that makes this possible.

The Problem: Why The Cloud Failed Netflix

To understand Open Connect, we have to understand the physics of video. We aren’t talking about serving JSON payloads or HTML pages. We are talking about streaming bytes of visual information—hundreds of thousands of them per frame.

If Netflix were a standard cloud deployment, every user request would have to travel across the public internet to a central data center. Here’s why that’s a death sentence for video:

The “Last Mile” Bottleneck: The public internet is a mess. Packets are routed through a maze of Tier 1 and Tier 2 networks, often taking circuitous paths across the globe. This introduces latency and packet loss, which kills the user experience (buffering) at scale.
Cost Explosion: Data transfer fees (egress) are the most expensive part of cloud computing. Paying AWS or Azure egress rates for petabytes of video would bankrupt even Netflix.
The “Thundering Herd” Problem: When a show like Wednesday drops, millions of people hit play within minutes. A centralized infrastructure would face a massive DDOS-like event induced by legitimate traffic.

The solution wasn’t to build a bigger data center. It was to move the data so close to the user that the “last mile” (the distance between the ISP and the home) became the only relevant network segment. The internet had to come to them.

Enter Open Connect: The ISP Within an ISP

Open Connect is Netflix’s proprietary CDN. But it’s not just a global network of servers; it’s a peering architecture designed to insert Netflix’s storage directly into the public internet’s fabric.

The core philosophy is: Don’t route around the traffic jam; build a tunnel right through it.

Netflix does this by deploying two specific types of appliances into the network:

1. The Open Connect Appliance (OCAs)

These are the workhorses. They are purpose-built, high-density storage servers running a customized version of FreeBSD (all tuned specifically for Netflix workloads). They live in colocation facilities alongside the ISP’s network.

Storage: These boxes can contain up to 100 TB of NVMe SSD storage or traditional spinning disks for colder content.
Function: They store the most popular content locally to serve the local community.

2. The “Virtual” OCAs (vOCA)

This is the interesting part. Netflix realized they couldn’t always place physical hardware where they needed it. So, they pivoted to virtualization.

Netflix runs a hypervisor on top of their OCAs that allows them to run multiple isolated virtual machines on a single physical host. This allows providers to offer Netflix services without hosting the dedicated hardware on their network edge. It increases density and allows for dynamic resource allocation.

But here’s the kicker: The appliance itself doesn’t just serve video. It runs a fully capable HTTP/2 and HTTP/3 (QUIC) server, TLS termination, and—crucially—the control plane that reports back to Netflix’s main AWS regions.

Key Insight: The OCA is not a passive cache. It is an active participant in the Netflix ecosystem that uses advanced algorithms to decide what content to store before you ask for it.

The Secret Sauce: Pre-Positioning and The “Fill” Strategy

Here is where “Open Connect” diverges from generic CDNs like Akamai. Most CDNs use a pull-through cache. A user requests a file, the edge cache misses, fetches it from the origin, serves it, and stores it for later.

Netflix hates cache misses. If the OCA doesn’t have the file, it has to fetch it from AWS, which introduces delay and—worst of all—uses expensive inter-DC bandwidth.

Instead, Netflix uses a pre-positioning strategy (also known as “fill”).

The Netflix Control Plane (in AWS)

Netflix still runs its intelligence and logic in AWS. They have a massive system called the Netflix Data Platform. When a new movie is uploaded, a complex series of microservices kicks in:

Ingestion: The master source file is uploaded to AWS S3.
Transcoding: AWS Auto Scaling groups launch hundreds of instances to transcode the video into multiple bitrates (ladders).
Analysis: The system identifies keyframes, scenes, and thumbnails.
Distribution: The transcoded files are pushed to “Origin” storage.

The Fill Engine

Here’s the magic. Netflix runs a service called the Fill API and the Fill Engine. Once a show is ready, this engine calculates a “replication factor.”

Global Hits: The hottest shows (like Bridgerton) are “filled” by pushing the content to every single OCA worldwide.
Regional Hits: Shows popular in a region (like a regional drama) are pushed to specific regions.
Long-Tail: Less popular content might be stored in only a handful of central locations or left to a “pull” request, but Netflix tries to avoid this as much as possible.

The Decision-Making Algorithm: How does Netflix decide what to put in a 100TB box?

They use a complex formula that weighs:

Popularity velocity: The rate at which a title is being requested.
Viewing preferences: Time-of-day patterns (watching movies at night, cartoons on Saturday mornings).
Hardware constraints: The storage capacity of the specific OCA they are targeting.

The Fill Engine schedules transfers during off-peak hours to avoid saturating backbone links. This means the data is waiting for you before you even click the play button.

The Hardware: When SSDs Meet Custom FreeBSD

Let’s talk hardware specs because this is where the “petabyte” scale truly hits home.

An OCA is not a 1U pizza box. These are often 4U servers packed to the gills with storage and networking gear. A typical high-density OCA looks like this:

CPU: Intel Xeon or AMD EPYC processors (to handle the massive TLS encryption workload).
Memory: 256GB to 512GB RAM (for hot caching and TCP stack optimization).
Storage: A full shelf of NVMe SSDs. Why NVMe over HDD? Because 4K streaming requires high IOPS for frequent read bursts, especially when serving many users simultaneously.
Network: 100Gbps Ethernet ports. They need to push gigabytes per second out to the ISP.

The “Netflix” FreeBSD

Netflix provides a customized version of the OS, called NetflixBSD. They aren’t using Linux. Why?

TCP Stack Control: FreeBSD’s network stack is highly modular. Netflix has patch-sets that allow them to implement custom congestion control algorithms (like their Centralized Concurrency Control).
Zero-Copy Networking: They leverage netmap and sendfile to move data from the NVMe disk to the network card with minimal CPU intervention. This means they can serve 10 Gbps of traffic with only 10% CPU utilization, leaving headroom for traffic spikes.

They also use TLS 1.3 with OCSP stapling to speed up handshakes. Since the content is static, they can pre-generate TLS tickets to avoid the expensive cryptography of a full handshake on every request.

The Peering Dance: How Netflix “Bribes” ISPs to Host Their Servers

This is the most fascinating part of the architecture: the business and networking logic.

Normally, a network serves a request and pays a Top-of-Rack switch cost and bandwidth costs. Netflix flips this upside down.

Netflix does not pay ISPs to host their boxes. They offer a deal: “Host our Open Connect Appliances for free, and we will give you the content for free, and we will take the load off your transit providers.”

The Value Proposition for the ISP

ISPs have two options:

Transit: Route user traffic to the internet, paying a Tier 1 provider like Lumen or Cogent per GB transferred.
Peering: Exchange traffic with Open Connect, which is a zero-cost interconnection.

When an ISP installs an OCA, they connect it directly to their edge routers. Now, when a subscriber requests Netflix, the traffic goes: Subscriber → ISP Router → OCA. It never leaves the ISP’s network.

For the ISP: They save millions in transit costs because they aren’t paying for the gigabytes.
For Netflix: They get to place their cache INSIDE the ISP, guaranteeing the highest Quality of Experience (QoE) with the lowest latency.

Network Tiering

Netflix runs a hybrid model:

Tier 1: ISPs with very high traffic volume get OCAs directly.
Tier 2: For smaller ISPs or regions where hardware isn’t feasible, Netflix connects via “Direct Peering” at major Internet Exchange Points (IXPs) like DE-CIX in Frankfurt.
Tier 3: If an ISP is too small for a peering agreement, Netflix might just rely on their “origin storage” in AWS, though this is rare for major markets.

The result is a distributed graph where Netflix controls the physical location of their data, bypassing the general chaos of the internet backbone.

The Engineering Behind the “Play” Button: The Request Path

When you press play, here is the exact sequence of events, mapping to the architecture we just discussed:

DNS Resolution: Your device asks the ISP’s DNS for api.netflix.com. The DNS server returns a PoP (Point of Presence) location, but not the server IP yet.
The Manifest: Your client requests a Manifest File (an XML/JSON file) from the Netflix control plane in AWS. This contains the URLs for the video chunks.
The Redirect: The manifest URLs point to a redirector service. This service uses network topology data to find the closest OCA to you.
The Fetch: The redirector returns a direct link to the OCA IP address. Your client initiates a TLS handshake directly with the OCA.
The Stream: The OCA uses its local disk to serve the MPEG-DASH chunks (segments of 2-6 seconds). It streams them over HTTP/2, pushing multiple segments in parallel to keep the buffer full.

This entire process—from clicking to the first frame—takes roughly 100-200 milliseconds. The actual video data never touches the AWS cloud once the content is filled.

The Scale: Crunching the Numbers

Let’s talk about the sheer size of this operation. As of 2024, Open Connect is responsible for delivering over 95% of Netflix’s global traffic.

Storage Capacity: Netflix announced they have over 200 Petabytes of SSD/NVMe storage embedded in OCAs globally. (That’s 200,000 Terabytes).
Throughput: During peak hours (typically 8-9 PM EST in the US), the CDN pushes terabits per second of traffic. A single OCA can saturate a 100Gbps link. Global peak is pushing towards the Exabit scale.
Server Count: Estimates place Open Connect at over 15,000 physical appliances worldwide.

To put that in perspective: Open Connect is likely the largest production deployment of FreeBSD in the world, and definitely the largest deployment of NVMe flash used for edge caching.

The Evolution: Moving to QUIC and HTTP/3

The CDN is not static; it is constantly evolving. Netflix is a pioneer in protocol innovation.

In the early days, they were heavy users of TCP. But TCP has a problem with mobile networks: Head-of-Line Blocking. If one packet gets lost, subsequent packets wait, even if they belong to a different part of the video.

Netflix has been rolling out QUIC (HTTP/3) aggressively. Why?

Faster Handshakes: QUIC combines the TLS handshake with the connection establishment, reducing connection time by up to 1 RTT.
Connection Migration: If your phone switches from Wi-Fi to 4G, the IP address changes. With TCP, this kills the connection. QUIC uses a Connection ID, so the stream survives the network switch without interruption.
Congestion Control: QUIC allowed Netflix to implement custom, in-userspace congestion control algorithms that can react to packet loss in milliseconds.

By deploying QUIC on their OCAs, they ensure that mobile users experience smooth playback even on congested cell towers.

The “Gingerbread” Monitoring Problem

Running a CDN of this magnitude requires monitoring at the edge.

They use a system called Gingerbread (yes, the cookie). Gingerbread is a telemetry system that runs on every OCA and sends performance metrics back to AWS.

QoE Metrics: They measure “Rebuffering Ratio” (the percentage of time the video buffers) and “Startup Time” for every session.
Detection: If a specific OCA starts showing high TCP retransmission rates (indicating a bad connection to the ISP), the intelligence in AWS automatically reduces the traffic sent to that OCA and redirects users to a peer OCA or a different path.

This is a closed-loop feedback system. The edge tells the core how healthy the edge is, and the core adjusts the routing.

The Curiosities: What Most Engineers Don’t Know

1. They Serve Content From Memory

For the top 1% of titles, the file is not read from disk on every request. OCA engineers use a page cache to keep the most popular files in RAM. Serving from RAM at 100Gbps is trivial for a CPU; serving from NVMe introduces latency. Netflix optimizes their cache logic to identify “hot” files quickly.

2. The “Peak Time” Migration

Netflix has a system that “evacuates” OCAs. If a specific data center is having a network issue or needs maintenance, they use a DNS-based steering to point all users to a different OCA. However, because OCAs are pre-filled, they can’t just be emptied easily.

They use a priority system: when the usage drops below a threshold, they “defill” the OCA by deleting the most popular content and re-allocating the storage. This is a continuous churn of data—writing, deleting, and rewriting petabytes of data daily.

3. They Actually Use HDDs for the Cold, Cold Content

The hype is all about SSDs, but Netflix is cost-prohibitive about everything. They mix storage classes. An OCA has 90% NVMe for active content, but 10% of the storage might be high-density HDDs for “archive” titles that nobody watches but have to be available. It’s a tiered storage hierarchy, just like in any database, but compressed into a physical box.

The Takeaway: You Can’t Just “Cloud” Your Way Out of Everything

The biggest lesson from Open Connect is the architectural philosophy: The edge is the database.

Netflix could have tried to scale their AWS fleet to handle the traffic. It would have been impossible to do cost-effectively.

Open Connect proves that when your product is massive and static (video), you must treat storage as a homing pigeon—send it where it needs to go before you need it. It’s the ultimate “push” architecture.

They solved the problem of “streaming petabytes seamlessly” not by finding a bigger pipe into their cloud, but by eliminating the cloud from the equation almost entirely. They moved the data to the user, buried their hardware inside the ISP’s infrastructure, and optimized every last packet to survive the hostile environment of the public internet.

The next time you press play and see the 100% progress bar fill up instantly, remember: you aren’t just accessing a video. You are tapping into a global network of purpose-built FreeBSD boxes, tuned beyond the bleeding edge, waiting in the dark, just for you.