You’ve seen the number: 3.2M watching. 8.7M watching. Then, during the 2023 Coachella livestream, the counter blinked past 100 million—and didn’t crash, didn’t lag, didn’t lie.
But here’s what you didn’t see: the silent war inside Google’s infrastructure. A war between eventual consistency and the brutal physics of live view counts. A war that ended with YouTube ripping out its beloved CRDTs (conflict-free replicated data types) and replacing them with a distributed state machine that tolerates 100 million concurrent watchers under 150ms latency.
This is the engineering autopsy of that transition. Buckle up.
The Context: When 3 Million Watchers Became a Lie
In 2020, YouTube’s live viewer count was a fire-and-forget CALM (Consistency as Logical Monotonicity) system. It used CRDTs—specifically a hybrid of state-based LWW-Registers (Last-Writer-Wins) and Delta-CRDTs—to aggregate view counts across 12 regional clusters. It was elegant. It was eventually consistent. And it was wrong.
Here’s the gut-churning reality:
- During high-profile live events (E3, World Cup finals, gaming premieres), the view count would oscillate wildly—jumping from 8M to 12M back to 9M within 30 seconds.
- The problem wasn’t network partitions. It was drift amplification. Each regional cluster would merge its local CRDT state with peer clusters, but the merge operations had non-idempotent retries due to clock skew. A single view event could be counted 4–7 times.
- The “100M” peak you see? In 2022, one event recorded 107M concurrent viewers, but internal audit logs showed 28% of those were phantom counts—duplicates created by CRDT anti-entropy passes that couldn’t distinguish between new viewers and replayed events.
The CRDT gospel preaches “no central coordinator, no conflict resolution.” But for live view counts? That’s a feature, not a bug. Users don’t want eventual consistency. They want instant consistency. They want the number to go up and stay up.
The Breaking Point: Why CRDTs Failed at Scale
Let’s get technical. YouTube’s old system used a Multi-Value Register pattern (MVR-CRDT). Every watcher session created a unique vector clock timestamped with a Google Global Clock (TrueTime) hybrid logical clock. The state was:
# Pseudocode of old CRDT merge
class ViewCountCRDT:
def __init__(self):
self.viewers = {} # {session_id: (TrueTime_epoch, node_id)}
def merge(self, other):
for session_id, (ts, node) in other.viewers.items():
if session_id not in self.viewers:
self.viewers[session_id] = (ts, node)
else:
# LWW: keep the later timestamp
if ts > self.viewers[session_id][0]:
self.viewers[session_id] = (ts, node)
self.count = len(self.viewers)This worked at 1M concurrent watchers. At 10M, the state object for a single live event ballooned to 2.4GB per replica because it stored every session_id as a key. The anti-entropy (gossip) protocol had to transfer diff states across 12 clusters every 5 seconds. Bandwidth cost: $47,000/hour for a single event.
At 100M, the system hit a wall:
- Compaction Hell: CRDTs require delta-intervals to garbage-collect old entries. But with 100M concurrent sessions, even the delta-state was 800MB. Broadcasting this across clusters caused TCP incast congestion on the border routers.
- Monotonicity Failure: A CRDT’s state must be monotonic (operations only add, never remove). But view counts must decrease when users leave. YouTube tried negative counter CRDTs—and discovered that merge operations cannot distinguish between “a user left” and “the network is slow, let me reapply the negative delta from two minutes ago.” The result: counts would temporarily go negative.
- Latency Jitter: The p99 latency for a view count update went from 47ms to 2.1 seconds under load because CRDT merges blocked on TrueTime synchronization between continents.
The Solution: Rebuilding as a Global State Machine
YouTube’s engineering team realized something radical: You don’t need CRDTs for this. You need a single, authoritative state machine—just distributed across the planet.
They built Titan, an internal global state machine framework, purpose-built for the live viewer count use case. It’s not a blockchain, not a consensus protocol like Raft, and not a database. It’s something in between: a partitioned state machine with quorum-based replication and speculative execution.
The Core Architecture
[User Watches Video]
|
v
Edge PoP (200ms TTL) --> Regional Aggregator (8ms window)
|
v
[**Titan Shard**] -- quorum write (3/5 replicas) --> State Machine
|
+--> In-memory HLL Counter (HyperLogLog) for tracking unique sessions
+--> Sliding window bitmap (100M bits) for active viewer dedup
+--> Write-ahead log (WAL) for crash recoveryKey engineering decisions:
1. Partitioned by Video ID + Hash Ring
Each live event gets a Titan shard containing exactly 5 replicas (spread across 3 GCP regions). The shard’s state machine processes all viewer count updates for that video. No CRDTs, no merge conflicts. One writer per shard (the leader), with 2-phase commit to followers.
2. HyperLogLog for Membership, Not Session IDs
Instead of storing session_id -> boolean, Titan stores HyperLogLog sketches of 64KB each. The HLL allows exact deduplication of viewers within a 2-second window, with 0.01% error. The memory per shard dropped from 2.4GB to 1.3MB for 100M watchers.
// Simplified Titan counter logic
type TitanViewCounter struct {
hll HyperLogLog // 14 bits precision (2^14 = 16384 registers)
window uint64 // Unix timestamp of current window
count uint64 // latest count
mu sync.Mutex
}
func (t *TitanViewCounter) Increment(userID uint64, timestamp uint64) error {
t.mu.Lock()
defer t.mu.Unlock()
// If timestamp crosses a 2-second window boundary, reset
if timestamp - t.window > 2000 {
t.window = timestamp
t.hll.Reset()
}
t.hll.Add(userID)
t.count = t.hll.Count()
return nil
}3. Sub-Second Latency via Speculative Reads
Here’s the dirty secret: clients don’t actually read from the leader. Every Edge PoP caches the last known count for 500ms. But to avoid showing stale data during peaks, the PoP performs a speculative read—it sends a “heartbeat ping” to the nearest Titan replica, and if the replica hasn’t advanced the leader’s state within 150ms, the PoP predicts the count using a linear regression model trained on the event’s view velocity.
This results in:
- p50 latency: 12ms (cached)
- p99 latency: 134ms (speculative)
- p99.9 latency: 890ms (force-synced from leader)
4. Crash Recovery with Deterministic Replay
If a shard leader dies, the new leader replays the WAL from the last checkpoint. But here’s the innovation: the WAL doesn’t store every increment—it stores only the HLL sketch state every 10 seconds. Replaying from a sketch is faster than replaying 100M individual increments. Recovery time: 1.2 seconds (down from 47 seconds with CRDTs).
The Hidden Complexity: Global View Count “Fairness”
You might think: “Why not just use a Redis counter per video?” Because YouTube’s live view count isn’t just a number—it’s a statement of fairness to creators.
Problem: During peak events, bots and manual refreshers inflate counts. YouTube needed to distinguish between:
- Organic viewers (one IP, one session, normal rate)
- Manual refreshers (hit F5 30 times/second)
- Bots (scale-coded attack)
Titan uses a 3-tier bloom filter:
| Tier | Window | Purpose | Memory per 100M watchers |
|---|---|---|---|
| Tier 1 | 1 second | Catch rapid refreshers (rate >10/sec) | 10KB |
| Tier 2 | 10 seconds | Catch suspicious rate (rate >30/10sec) | 100KB |
| Tier 3 | 5 minutes | Block sustained bot attacks | 10MB |
If a session hits Tier 1, it’s soft-dropped (counted but not added to the public counter). If it hits Tier 2, it’s hard-dropped (ignored entirely). The bloom filters are shared across shards via a gossip protocol that replicates only the filter’s hash, not the session IDs.
The Infrastructure Behind 100M Watchers
YouTube runs Titan on GKE nodes with custom-built NIC kernels. The critical path:
Edge PoP (200 locations)
|
v
Google Front End (GFE) -- maps video to shard
|
v
**Titan Proxy** (in-process sidecar) -- dedup, rate limit, bloom filter
|
v
**Titan Shard** (5 Pods, 3 regions):
- Pod 0: Leader (handles writes)
- Pod 1-4: Followers (handle reads)
- Shared disk: Local SSD (NVMe, 10μs latency)
- WAL: Google Cloud Spanner (for global durability)The hardware per Pod:
- CPU: 96 cores (2x AMD EPYC 7B12), >50% reserved for JVM GC pauses
- Memory: 512GB RAM (300GB reserved for HLL sketches + Bloom filters)
- Network: 200 Gbps via Google’s Jupiter network (custom optical fabric)
- Disk: 4x 15TB NVMe SSD in RAID 0 (WAL writes at 1.2M IOPS)
The cost per live event:
- Compute: $0.04 per 1,000 watchers per hour
- Network: $0.01 per 1,000 watchers (due to Titan’s optimized gossip reducing broadcast from 10MB/s to 120KB/s per shard)
The Results: What Changed for Users?
Before Titan (CRDT era):
- View count accuracy: ±12% (28% phantom during major events)
- p99 latency to see your own view: 2.3 seconds
- Max sustained concurrent watchers: 12 million
After Titan (State Machine era):
- View count accuracy: ±0.8% (real, auditable against ad-server logs)
- p99 latency to see your own view: 134ms
- Max sustained concurrent watchers: 107 million (peaked at 114M during a single event)
The Engineering Lesson: Consistency Is a Product Choice
YouTube chose to move away from perfect CRDT theory and toward a pragmatic state machine. Here’s the hard truth: CRDTs are beautiful for collaborative editing (Google Docs) or DNS-like systems (where monotonicity is baked in). But for view counts, where users demand instant, correct, and monotonic numbers, a state machine with deterministic replication is simpler, faster, and cheaper.
The takeaway: When you hit 100M concurrent users, you don’t need to sync every mouse click. You need to deduplicate, speculate, and lie (just a little) about the truth. Titan’s speculative reads are mathematically proven to never overcount by more than 0.1% —and users feel the speed more than they notice the inaccuracy.
The Future: Titan’s Next Generation
YouTube is already testing Titan v2, which will:
- Replace HLL with CVM (Consistent Virtual Meshes) for zero-error counting
- Use FPGA-based packet processing to run bloom filters in the optical layer (no CPU skip)
- Implement locality-aware sharding: if 80% of viewers are in India, Titan will move the shard leader to Mumbai automatically
The dream: 1 billion concurrent watchers, with 100ms latency, and zero duplicates.
Final Thought: The Number You See Is a Lie—But a Good One
The view count on your screen isn’t the actual number of viewers. It’s the output of an incredibly complex global state machine that has to trade off between freshness and accuracy. YouTube’s Titan system chooses to occasionally show you a number that’s 0.1% too high, rather than 0.1% too old. And that’s okay.
Because for the 100 million people watching the same thing at the same time, the number isn’t just a metric. It’s a shared moment. And that moment can’t wait for CRDT convergence.
Enjoyed this? Drop a comment below—I’ll try to respond within 134ms.