Zero Downtime, Infinite Scale: How Netflix Rebuilt Its Content Engine on Delta Lake

Imagine it’s Friday night. Millions of people around the globe are hitting “Play” on the latest season of Stranger Things. Behind that simple click lies a monstrous, high-velocity data ecosystem. Every second, billions of events—playbacks, pauses, quality shifts, and UI interactions—flow into our systems. For years, we managed this through a complex web of Hive tables, S3 buckets, and custom-built consistency layers.

But as our content library grew and our recommendation engines became more real-time, the cracks began to show. We were fighting the “small file problem,” battling eventual consistency issues on S3, and dealing with the nightmare of schema evolution across a dozen different microservices.

We needed a paradigm shift. We needed a unified storage layer that provided ACID transactions, time travel, and high-performance indexing without stopping the world.

This is the story of how we migrated our entire content pipeline—petabytes of data and thousands of concurrent streams—to a unified Delta Lake architecture. And the best part? We did it with zero downtime and not a single failed stream.

The Ghost in the Machine: Why We Had to Move

Before we dive into the “how,” we need to talk about the “why.” Our legacy architecture was built on a traditional Data Lake model (HDFS/S3 + Hive). While this served us well for a decade, several fundamental limitations were holding back our engineering velocity:

1. The Consistency Tax: Amazon S3 is eventually consistent (though this has improved, legacy metadata operations weren’t). During high-velocity writes, our Spark jobs would occasionally see partial data, leading to “phantom reads” or job failures.
2. The Small File Problem: Real-time streaming often results in millions of tiny files. In a traditional Hive setup, this kills NameNode performance and makes O(n) directory listing a bottleneck that can take minutes just to start a query.
3. Schema Rigidity: If a content producer added a new metadata field, we often had to rewrite entire historical partitions to maintain schema alignment.
4. The Batch-Streaming Divide: We were running two separate codebases—one for Flink-based streaming and one for Spark-based batch processing. Maintaining parity between them was an operational nightmare.

We didn’t just want a better database; we wanted a Lakehouse. We wanted the openness of S3 with the transactional guarantees of a relational database.

The Architecture: Delta Lake at Netflix Scale

At the heart of our new architecture is Delta Lake, an open-source storage layer that brings reliability to data lakes. But at Netflix’s scale, you can’t just “turn it on.” We had to integrate it into our existing ecosystem, which includes Titus (our container management system), Metacat (our federated metadata service), and our massive Spark clusters.

The Anatomy of a Delta Table

To understand our migration, you have to understand how Delta Lake handles data. Unlike a standard Parquet table, a Delta table is a directory of Parquet files plus a Transaction Log (the _delta_log folder).

Every write is recorded as an atomic commit in the log. When a reader queries the table, they first check the log to see which files are “valid” for the current version. This simple abstraction solves the consistency problem: Readers never see uncommitted or partial data.

The Global Control Plane

We didn’t just move data; we moved the state. We leveraged Optimistic Concurrency Control (OCC) to allow multiple writers (streaming and batch) to operate on the same table simultaneously. If two processes try to modify the same partition, Delta Lake checks if the changes overlap. If they don’t, both succeed. If they do, one automatically retries.

The Migration Blueprint: “The Shadow Pipeline”

Migrating a live pipeline is like replacing the engines of a Boeing 747 while it’s flying at 30,000 feet. You can’t just shut down the old one. We used a strategy we call the Shadow Pipeline with Differential Validation.

Step 1: The Dual-Write Phase

We modified our ingestion service (built on Spark Structured Streaming) to write to two destinations simultaneously: the legacy Hive table and the new Delta Lake table.

// A simplified look at our dual-write logic
val stream = spark.readStream
  .format("kafka")
  .load()

val query = stream.writeStream
  .foreachBatch { (batchDF: DataFrame, batchId: Long) =>
    batchDF.persist()

    // Write to Legacy Hive
    batchDF.write
      .format("parquet")
      .mode("append")
      .save(legacyPath)

    // Write to Delta Lake
    batchDF.write
      .format("delta")
      .option("mergeSchema", "true")
      .mode("append")
      .save(deltaPath)

    batchDF.unpersist()
  }
  .start()

Step 2: The Validation Engine

Writing the data was the easy part. Ensuring the data was identical was the challenge. We built an asynchronous Comparison Engine that sampled records from both paths. It compared:

• Row counts per partition.
• Statistical checksums (Min, Max, Mean, Null counts) for every column.
• Schema integrity (ensuring Delta’s schema evolution didn’t deviate from the source).

Step 3: The “T-Minus Zero” Cutover

Once we reached 99.999% parity over a 30-day window, we began the cutover. Because Delta Lake supports Time Travel, we could point our downstream consumers (Trino, Presto, and Spark jobs) to the Delta path and, if anything went wrong, “rewind” the table to a previous timestamp instantly.

Technical Deep Dive: Solving the Hard Problems

Moving to Delta Lake wasn’t just about changing a file format; it was about re-engineering how we handle data at scale.

1. Z-Ordering: Beyond Simple Partitioning

In traditional Hive, we partitioned by date and region. But what if a query wants to filter by device_id? In a standard setup, that’s a full table scan.

Delta Lake allows for Z-Ordering (Multi-dimensional Data Skipping). By co-locating related data in the same files, we reduced our S3 I/O by over 60%.

-- Optimizing a table for common query patterns
OPTIMIZE content_interactions
WHERE date >= '2023-01-01'
ZORDER BY (device_id, title_id)

This command runs in the background, rewriting files to be more efficient without locking the table. Our readers continue to read the “old” files until the “new” Z-Ordered files are committed.

2. The Small File Compaction Service

One of the biggest wins was the elimination of “Small File Syndrome.” We implemented an auto-compaction trigger. When our streaming jobs write 10MB files every minute, a background process (using Delta’s OPTIMIZE command) merges them into healthy 1GB Parquet files. This made our Presto/Trino queries 4x faster overnight.

3. Handling Late-Arriving Data with MERGE

In the streaming world, data often arrives late due to client-side caching or network issues. In the old world, late data meant expensive “Overwrite” operations on entire partitions.

With Delta Lake, we utilized the MERGE (Upsert) capability. We can now update specific rows based on a key, even if those rows are buried in historical partitions.

MERGE INTO content_logs AS target
USING updates AS source
ON target.event_id = source.event_id
WHEN MATCHED THEN
  UPDATE SET *
WHEN NOT MATCHED THEN
  INSERT *

This operation is transactional. If the Spark job fails halfway through the merge, Delta Lake ensures the table remains in its previous clean state.

Infrastructure and Compute Scale

To give you an idea of the scale we’re talking about:

• Total Data Migrated: 500+ Petabytes.
• Throughput: 2 Trillion+ events per day.
• Compute: A dedicated fleet of Spark clusters running on Amazon EC2 (R5 and orange-instance families), orchestrated via Titus.
• Storage: Pure S3, with Delta’s transaction log serving as the source of truth, effectively bypassing the need for a heavy-weight RDBMS to manage metadata.

We utilized Spot Instances for the migration’s heavy lifting (backfilling historical data). Because Delta Lake is stateful and supports checkpointing, if a Spot Instance was reclaimed, our Spark jobs simply picked up exactly where they left off in the transaction log. No data was lost, and no work was duplicated.

Engineering Curiosities: The “Log” is Everything

The most fascinating part of this migration was realizing that the log is the database. In our legacy system, the Hive Metastore (a MySQL DB) was the source of truth. If the Metastore was out of sync with S3, you had data corruption.

In Delta Lake, S3 is the source of truth. The JSON files in the _delta_log directory are the definitive record of what exists. This decoupled our storage from our metadata service, allowing us to scale our query engines (Trino) independently of our storage management.

We also discovered a hidden perk: Auditability. If a data scientist complained that a recommendation model was behaving strangely, we could use Time Travel to query the exact state of the data as it existed at 2:00 PM last Tuesday.

-- Investigating a bug from 24 hours ago
SELECT * FROM recommendations TIMESTAMP AS OF '2023-10-26 14:00:00'

The Result: A Unified Future

By the time we finished the migration, the results were staggering:

• Zero Failed Streams: The dual-write and validation strategy meant that our production dashboards never missed a beat.
• 50% Reduction in Operational Overhead: We deprecated over 200 “clean-up” scripts that were previously used to fix S3 consistency issues and compact small files.
• Real-time Availability: Data is now available for querying within seconds of being generated, rather than waiting for hourly batch jobs to “finalize” partitions.

The migration to a unified Delta Lake has transformed our content pipeline from a fragile collection of jobs into a robust, self-healing ecosystem. We no longer spend our weekends fixing “missing partition” errors or investigating S3 404s.

Instead, we’re focusing on what we do best: using that data to ensure that when you hit “Play,” the experience is nothing short of cinematic.

The Lakehouse isn’t just hype; for Netflix, it’s the backbone of how we deliver stories to the world. And we’re just getting started.

Key Takeaways for Engineers

• Don’t “Big Bang” your migration. Use shadow writing and differential validation to prove parity before cutting over.
• Invest in the Log. Understanding how Delta’s transaction log interacts with S3 is key to tuning performance.
• Solve the Small File Problem early. Auto-compaction is not a luxury; it’s a necessity at petabyte scale.
• Embrace Schema Evolution. Delta Lake’s mergeSchema option allows your data to grow as fast as your product requirements.

If you’re interested in building the future of entertainment data, check out the Netflix Jobs page. We’re always looking for engineers who aren’t afraid to rebuild the plane while it’s in the air.