System Design #4 Hello Interview System Design Guide (Evan King) 10 Billion Pages in 5 Days • 23,150 Pages/Sec • LLM Pre-Training Corpus

Design a Distributed Web Crawler for LLM Pre-Training

Production-grade decoupled multi-stage web crawler: URL Frontier (SQS), 8x 200Gbps Fetcher fleet with in-memory Unbound DNS caching and multi-provider round-robin, atomic Redis politeness delay locks with anti-synchronization jitter, S3 Raw HTML blob store (preventing message queue bloat), Further Processing SQS queue, auto-scaling Text & URL Parser workers, S3 Clean Text dataset, content-hash deduplication (Metadata DB index vs 14.4GB RedisBloom), and hop-depth crawler trap guards.

SLA: 23,150 Pages/Sec (3.9 Days)8x c6in.32xlarge • SQS DLQ (5 Retries)

Aggregate Crawl Rate

23,150pages/s

10B Pages / 5 Days = 23,148 req/s

Network Bandwidth

18.5Gbps

8x 200 Gbps AWS Instances (30% util)

Frontier SQS Backlog

5URLs pending

Lightweight URL IDs (No HTML)

Duplicates Filtered

6,120pages dropped

DB Hash Index vs 14.4GB RedisBloom

Local DNS P99 Latency

0.48msP99 in-mem

Unbound Cache + Multi-Provider DNS

Simulate Pipeline:

SQS Frontier (URL IDs) 8x Fetchers & DNS Cache S3 Stores & Parser Fleet Dedupe (DB Index / RedisBloom) Hop-Depth Guard (Max 15) SQS DLQ (5 Retries)

Click any component box to view Hello Interview architectural deep dive & tradeoffs

URL Frontier SQS (Pending IDs)

5 URLs

https://en.wikipedia.org/wiki/Large_language_modelDomain: en.wikipedia.org • Hop Depth: 0 • Retries: 0

https://arxiv.org/abs/2303.08774Domain: arxiv.org • Hop Depth: 0 • Retries: 0

https://github.com/trendingDomain: github.com • Hop Depth: 0 • Retries: 0

https://nytimes.com/section/technologyDomain: nytimes.com • Hop Depth: 0 • Retries: 0

https://en.wikipedia.org/wiki/Transformer_(deep_learning)Domain: en.wikipedia.org • Hop Depth: 1 • Retries: 0

Amazon S3 Storage Buckets

s3://llm-training-corpus/en.wikipedia.org/LLM.txt.gz12:00:02 • Hash: 8f7a2b • Target: clean_text

104.2 KB

s3://llm-training-corpus/arxiv.org/2303.08774.txt.gz12:00:03 • Hash: c9d4e1 • Target: clean_text

88.6 KB

Crawl Console & Dedupe Log

[12:00:01]System initialized: SQS Frontier, 8x Fetcher Pods, Unbound DNS, and S3 Storage online.

Core Requirements (Hello Interview Spec)

System Goal: Crawl the web to extract raw text data for training an LLM (e.g. OpenAI GPT-4, Google Gemini, Meta LLaMA).

Functional 1: Crawl the web starting from given seed URLs, recursively extracting text & following hyperlinks.

Functional 2: Extract clean text and store in S3 blob storage for subsequent model processing.

Non-Functional Scale: 10 Billion pages in under 5 days (~23,150 pages/sec aggregate throughput).

Politeness & Fault Tolerance: Respect robots.txt & 1 req/s rate limits per host; recover from crawler crashes without losing progress using SQS visibility timeouts.

System Interface & Scope Boundary

System Interface:

• Input: Curated Seed URLs (top directories, Wikipedia, major news sites)
• Output: Text data extracted from web pages stored in S3

Below the Line (Out of Scope):

Actual LLM model pre-training / tokenization
Non-text data (video, audio, binary attachments)
Scraping 100% of the entire internet (acknowledging small dark corners are unreachable)
Authenticated pages requiring user login credentials

Anti-Pattern Avoidance:Never store raw HTML payloads inside SQS/Kafka queues! Queues only pass URL IDs; HTML is written directly to S3.

System Design Deep Dives & Technical Reference

Comprehensive architectural breakdown based on Evan King's HelloInterview guide.

Back-of-the-Envelope Calculations (10 Billion Pages in 5 Days)

In system design interviews, Evan King recommends delaying BOE calculations until solving a specific problem. Here is how we mathematically size the crawler fleet:

1. Target Ingestion Throughput23,150 pages / sec10,000,000,000 / (5 days × 86,400s) = 23,148 req/s

2. Machine Sizing (AWS c6in.32xlarge)8 Instances (3.9 Days)200 Gbps × 30% util ÷ 2MB = 3,750 pps → 8 nodes

3. Total Text Storage~1.0 Petabyte10B pages × ~100 KB compressed clean text

💡 The 23,150 pps vs 1 req/s Paradox:

• How can we crawl 23,150 pages/s when rate-limited to 1 request/second per host?

• Key Insight: We are crawling millions of distinct domains in parallel! Each worker maintains thousands of concurrent TCP/TLS connections to different websites simultaneously.

URL Frontier Queue (Amazon SQS)

Role & Overview

The queue holding URLs waiting to be fetched. Built with Amazon SQS to provide managed elastic scaling, native visibility timeout mechanisms, and automatic Dead-Letter Queue (DLQ) support.

Architecture Implementation

1. Messages contain ONLY the URL ID / reference to Metadata DB (never raw HTML!). 2. When worker receives message, visibility timeout hides it from other workers. 3. Failed fetches trigger ChangeMessageVisibility with exponential backoff.

Frontier Queue Technology Trade-offs

❌ Bad Solution: In-Memory Timer: Waiting a few seconds in memory fails if the crawler crashes, losing timer and state.

⚠️ Good Solution: Apache Kafka with Manual Retries: Retains log offsets, but requires implementing a custom retry topic and scheduled consumer delay.

⭐ Great Solution: Amazon SQS with Visibility Timeout: Native ChangeMessageVisibility dynamically calculates backoff as base_delay * 2^(recv - 1), automatically routing to DLQ after 5 retries.

Specification & Code Snippet

// SQS Dynamic Visibility Timeout Exponential Backoff
async function handleFetchFailure(message) {
  const receiveCount = parseInt(message.Attributes.ApproximateReceiveCount);
  if (receiveCount >= 5) {
    // SQS Redrive Policy automatically moves message to DLQ
    await markSiteOffline(message.Body.host);
    return;
  }
  // Exponential backoff: 30s, 60s, 120s, 240s...
  const visibilityTimeout = Math.min(30 * Math.pow(2, receiveCount - 1), 43200);
  await sqs.changeMessageVisibility({
    QueueUrl: FRONTIER_QUEUE_URL,
    ReceiptHandle: message.ReceiptHandle,
    VisibilityTimeout: visibilityTimeout
  }).promise();
}