V
โ† Back to System Design Showcase

๐Ÿ† Design Top-K Video Views at Scale (YouTube Heavy Hitters)

5B+ Views/Day โ€ข 1M+ Views/sec Peak โ€ข Count-Min Sketch โ€ข Flink โ€ข Redis ZSET โ€ข ClickHouse

Real-time heavy hitters architecture across 1 Hour, 1 Day, 1 Month, and 1 Year horizons. Combines Count-Min Sketch frequency estimation with a bounded Min-Heap (K=10), two-phase scatter-gather aggregation against viral hotspots, rolling 60-minute circular Redis ZSET buffers, ClickHouse SummingMergeTree long-term rollups, and offline Spark self-healing reconciliation.

Window:
Traffic:
Ingestion Write QPS
520,000views/s
128 Kafka Partitions Active
Top-K Read Latency
1.8msp99
CDN Edge: 3ms โ€ข Redis: 1.1ms
Sketch RAM Footprint
760 KBbounded
w=27,183 โ€ข d=7 (ฮต=0.01%)
Active Heap Entries
10 (Full)candidates
K=10 Top Videos in State
Reconciliation Drift
0.02%
Ground Truth vs Streaming

๐ŸŒ Top-K Real-Time Stream & OLAP Architecture

Client Ingestion โ†’ Two-Phase Kafka Partitioning โ†’ Flink Count-Min Sketch โ†’ Bounded Min-Heap โ†’ Redis ZSET Hot Cache โ†’ ClickHouse OLAP โ†’ Edge CDN.

System Ready โ€ข Click any component node to inspect design trade-offs
COL 1: INGESTIONCOL 2: KAFKA BUFFERCOL 3: FLINK STREAMCOL 4: CMS & HEAPCOL 5: STORAGE (REDIS/OLAP)COL 6: SERVING & CDNClient Video PlayerPlayback HeartbeatPOST /views/heartbeatWatch โ‰ฅ 30s CriteriaView Validator & Bot GuardBloom Filter & Token BucketDrops reload spamLegitimate Views OnlyKafka View StreamTwo-Phase Partition Keyingvideo_id + salt(0..15)128 Partitions โ€ข 1M QPSFlink Stream EngineTwo-Phase Scatter Gather10s Micro-BatchesRocksDB CheckpointsCount-Min Sketch (CMS)Probabilistic 2D Matrixw=27,183 ร— d=7 Counters760 KB RAM โ€ข ฮต=0.01%Bounded Min-Heap (K=10)Heavy Hitter SelectorO(log K) Root EvictionStrict Top 10 OrderRedis ZSET Hot Store60m Circular Ring Bufferviews:top:1h โ€ข views:top:24hZREVRANGE < 1.5msClickHouse OLAPSummingMergeTreeMonthly & Yearly RollupsColumnar AggregationTop-K Query APISingle-Flight GuardMetadata Hydration (Titles)GET /api/v1/videos/topEdge CDN (Fastly / CF)Global Edge Cachingmax-age=10s, stale=5s99.8% Cache Hit RateOffline Spark ReconcilerSelf-Healing Ground TruthNightly Raw S3/GCS AuditHeals Drift > 0.5%
Idle
Click any component card in the SVG to inspect architectural trade-offs.
Client & Ingestion
Streaming & CDN
Count-Min Sketch (CMS)
Min-Heap (K=10)
Redis ZSET Hot Cache
๐Ÿ‘† Click any component in the architecture diagram to open its floating interview details.

Core Requirements (The Hello Interview Breakdown)

Functional Requirements:
  • 1. Real-Time Top-K: Return the Top 10 most-viewed videos across 1 Hour, 1 Day, 1 Month, and 1 Year windows.
  • 2. Dimensional Filtering: Support breakdown by Country (US, IN, Global) and Category (Music, Gaming, News).
  • 3. View Qualification: Filter out bot fraud, invalid playback sessions, and views under 30 seconds.
Non-Functional Requirements:
  • 1. Ultra-Low Read Latency: P99 latency < 20ms globally for the trending leaderboard.
  • 2. High Ingestion Throughput: Ingest 500k+ views/sec peak without dropping valid playback events.
  • 3. Bounded Memory: Stream processing must use sublinear O(log N) space rather than tracking billions of IDs.
  • 4. Eventual Consistency & Self-Healing: Real-time approximate leaderboards reconciled against batch ground truth.
Scale: 5B+ Views/Day โ€ข 1M+ Views/sec PeakP99 Read Latency: < 20ms (CDN < 3ms)

What is Expected at Each Level?

Mid-Level: Design a basic stream pipeline: Ingest views via Kafka, aggregate with a sliding window in Flink or Redis, and store Top-K in a Redis Sorted Set (ZSET). Explain how to query ZREVRANGE.
Senior: Explain why exact counting fails at 1M views/sec and detail the Count-Min Sketch + Min-Heap algorithm. Address hot partitions via Two-Phase Scatter-Gather. Design the hierarchical rollup storage (Redis ZSET for Hour/Day, ClickHouse SummingMergeTree for Month/Year).
Staff+: Architect end-to-end multi-dimensional partitioning across Country and Category without combinatorial explosion. Formulate the offline Spark reconciliation loop to self-heal approximation drift. Detail single-flight cache stampede protection and evaluate system sizing for 5B daily views.
Count-Min Sketch (CMS)
Role & Responsibility

Sublinear Space Frequency Estimator

Description

2D array of counters (width w, depth d) paired with d independent hash functions. Estimates the frequency of billions of videos using only 760 KB of RAM.

Production Implementation

1. Width w = ceil(e / epsilon) = 27,183 (for 0.01% error). 2. Depth d = ceil(ln(1 / delta)) = 7 (for 99.9% confidence). 3. On view arrival: increments d counters; estimated count = min(row_0..row_d).

Architectural Trade-Off: Hash Table vs Count-Min Sketch
โŒ Bad Approach: In-Memory HashMap: Tracking 1 billion videos requires 32 GB RAM and causes catastrophic Stop-the-World garbage collection pauses.
โญ Staff-Level Solution: Count-Min Sketch (w=27,183, d=7): Requires exactly 760 KB RAM, never runs out of memory, and guarantees frequency overestimation <= epsilon * N with 99.9% probability!
Specification & Code Snippet
// Count-Min Sketch Algorithm
type CountMinSketch struct {
    w, d    int
    matrix  [][]uint32
    hashes  []HashFunc
}
func (cms *CountMinSketch) Update(videoID string, count uint32) uint32 {
    minEst := uint32(math.MaxUint32)
    for i := 0; i < cms.d; i++ {
        idx := cms.hashes[i](videoID) % uint32(cms.w)
        cms.matrix[i][idx] += count
        if cms.matrix[i][idx] < minEst {
            minEst = cms.matrix[i][idx]
        }
    }
    return minEst // Frequency guarantee: TrueCount <= minEst <= TrueCount + ฮตN
}