📈 Design Metrics Monitoring & Alerting at Scale

10M+ Metrics/sec • Gorilla TSDB (Delta-of-Delta) • PromQL • Flink Streaming

Distributed observability architecture: Monitored Microservices emitting counters/histograms, hybrid pull scrapers & OTLP receivers, Kafka partitioning buffer, in-flight Flink stream pre-aggregation, Prometheus TSDB with Gorilla compression (91.4% memory reduction), AlertManager deduplication/inhibition, and tiered S3 downsampling.

Mode:

Ingestion Throughput

10.4Mpts/s

● 128 Kafka Partitions Active

TSDB Write Latency

1.2msp99

Append-only NVMe WAL

Gorilla TSDB Footprint

1.37 B/ptbytes/pt

91.4% memory compression

Evaluated Alert Rules

1,420/ 15s

● 0 Firing Incidents

Active Time Series

48.2Mseries

Cardinality guard: 0 drops

Architecture Canvas(Click any node to inspect interview deep-dive solutions)

Ingress & Discovery

Buffer & Flink

Gorilla TSDB

Alert & Query

📋

Core Requirements & Scope

Functional Requirements

1. Metric Ingestion: Ingest counters, gauges, and histograms from 100k+ microservices via both pull scraping and push gateways.
2. Low-Latency Querying: Execute complex multi-dimensional PromQL / M3QL queries across arbitrary time ranges with sub-second response.
3. Anomaly Alerting & Routing: Continuously evaluate alerting rules every 15s, group related incidents, suppress cascades, and route notifications to PagerDuty/Slack.
4. Multi-Tier Retention: Retain high-resolution raw data in memory/SSD for 7 days, downsample to 1m/1h aggregates, and archive to cold S3 for 1+ years.

Non-Functional Requirements & SLA

1. Ultra-High Write Scalability: Handle 10M+ metrics/sec sustained without data loss or backpressure.
2. High Availability: 99.99% uptime for ingestion; buffer bursts via Kafka during downstream TSDB maintenance.
3. Cardinality Safety: Automatically detect and drop high-cardinality label explosions before they destabilize the TSDB index.
4. Cost Efficiency: Achieve >90% compression ratio on stored samples using Gorilla encoding and automated block compaction.

Out of Scope

Distributed tracing span collection (OpenTelemetry Traces / Jaeger)
Centralized full-text log aggregation (Elasticsearch / Loki)
Automatic runtime bytecode APM profiling

Source: Hello Interview System Design GuideScale: 10M+ Metrics/sec

🎯

What is Expected at Each Level?

Mid-Level (L4 / IC4)Core Architecture

Understands TSDB vs Relational DB trade-offs, explains why standard SQL tables crash under 10M inserts/sec, designs basic pull vs push ingestion, and configures AlertManager routing.

Senior (L5 / IC5)Performance & Resiliency

Designs Gorilla TSDB delta-of-delta timestamp and XOR float encoding (achieving 1.37 bytes/sample), sets up Kafka partition key strategy by composite hash(tenant, metric), builds Flink tumbling window rollups, and implements AlertManager deduplication/inhibition.

Staff+ (L6 / IC6+)Enterprise Scale & Edge Cases

Mitigates catastrophic high-cardinality explosions with real-time HyperLogLog token buckets, architects a 3-tier storage lifecycle (NVMe Head → 5m Compaction → S3 Archival) saving 95% cost, and designs federated zero-downtime TSDB query planners (Thanos/M3DB pattern).

Interview Standard: Senior / Staff Expectations99.99% Availability

Prometheus TSDB Head Chunks (Gorilla Engine)

Role & Overview

High-performance time-series database engine running on NVMe nodes. Uses Gorilla delta-of-delta timestamp compression and XOR floating point encoding to store millions of data points in RAM.

Architecture Implementation

1. Head Chunks hold recent 2 hours of live samples entirely in memory for instantaneous reads. 2. Double-delta timestamp encoding compresses 32-bit timestamps down to an average of 1.37 bits. 3. Inverted index (Postings lists) maps tag key=value pairs to numeric time series IDs using roaring bitmaps.

Gorilla Compression Algorithm Deep Dive

❌ Bad Solution: Store Timestamps as 64-bit Ints & Values as 64-bit Floats: 16 bytes per sample. 10M samples/sec requires 13.8 TB RAM per day!

⚠️ Good Solution: Snappy / Gzip Chunk Compression: Compresses data blocks to ~4 bytes/sample, but requires decompressing the entire block to read a single value.

⭐ Great Solution: Facebook Gorilla TSDB Encoding: Delta-of-delta timestamps (usually 0 bits) + XOR floating-point values. Reduces footprint to ~1.37 bytes/sample (91.4% memory savings) with streaming bit-level random access!

Specification & Snippet

// Gorilla Delta-of-Delta Timestamp Encoding
func EncodeTimestamp(t, prevT, prevDelta int64) (bits uint64, numBits int) {
    delta := t - prevT
    dod := delta - prevDelta
    if dod == 0 {
        return 0, 1 // Exactly on schedule: 1 bit ('0')
    } else if dod >= -63 && dod <= 64 {
        return (0b10 << 7) | uint64(dod+63), 9 // 9 bits
    } else if dod >= -255 && dod <= 256 {
        return (0b110 << 9) | uint64(dod+255), 12 // 12 bits
    } else {
        return (0b1111 << 32) | uint64(dod), 36 // Full 36 bits
    }
}