🤖 Design ChatGPT (LLM Serving Platform)

100M+ DAU • vLLM Continuous Batching • PagedAttention • SSE Streaming

High-scale LLM serving architecture: Client Server-Sent Events (SSE) streaming, semantic vector caching, moderation guardrails, continuous iteration-level batching, PagedAttention KV cache VRAM pooling, and 8-way Tensor Parallel GPU inference.

Model:

Active SSE Streams

10,450concurrent

Continuous Batching

Time to First Token (TTFT)

280msP99

Prefill: 180ms

Generation Throughput

74.2tok/s/user

680k tok/s cluster-wide

KV Cache Waste

3.8%VRAM

PagedAttention Active

Semantic Cache Hit

24.1%ratio

Saves ~24% GPU Compute

🌐 LLM Inference Topology

Client SSE → Gateway Rate Limiter → Moderation → Orchestrator → Semantic Cache → vLLM Router → PagedAttention → 8x H100 Cluster.

System Ready • Click any node to inspect

Idle

Click any component card in the SVG to inspect architectural trade-offs.

Client & Ingress

Safety Moderation

Semantic Cache

vLLM & H100 GPUs

👆 Click any component in the architecture diagram to open its floating interview details.

Core Requirements (The Hello Interview Breakdown)

Functional Requirements:

1. Real-Time Streaming: Stream generated text tokens to user as they are synthesized.
2. Multi-Turn Context: Maintain conversational history across multiple questions and answers.
3. Safety Moderation: Filter harmful, toxic, or prompt-injection inputs prior to model inference.

Non-Functional Requirements:

1. Ultra-Low First-Token Latency: TTFT < 300ms to provide immediate responsive feedback.
2. High GPU Throughput: Maximize tokens/sec/dollar and eliminate GPU idle bubbles.
3. Memory Efficiency: Eliminate KV cache VRAM fragmentation (PagedAttention).
4. Scalability & Availability: Autoscale across multi-node GPU clusters with 99.99% availability.

Scale: 100M+ DAU, 10k Concurrent StreamsTTFT: < 300ms • 70 tokens/sec

What is Expected at Each Level?

Mid-Level: Design the end-to-end flow: Client SSE streaming, API Gateway authentication, chat context truncation, and basic model invocation.

Senior: Explain how Continuous (Iteration-Level) Batching prevents GPU starvation compared to static batching. Detail PagedAttention KV cache virtual memory management and Semantic Vector Caching with embeddings.

Staff+: Deep dive into multi-node Tensor Parallelism vs Pipeline Parallelism over NVLink/InfiniBand. Architect Speculative Decoding and evaluate system sizing for 100M DAU ($50M+ GPU infrastructure budget). Formulate fault tolerance, dynamic quantization (FP8/INT4), and multi-region failover under GPU cluster hardware failures.

Tensor-Parallel GPU Cluster

Role & Overview

NVIDIA H100 GPU nodes interconnected via 900 GB/s NVLink. Splits large model weights (e.g. Llama-3 70B / 405B) across 8 GPUs using Tensor Parallelism.

Architecture Implementation

1. 8-way Tensor Parallelism (TP=8): Splits Multi-Head Attention and MLP matrices across 8 GPUs in a node. 2. All-Reduce over NVLink: Synchronizes intermediate activations in microseconds. 3. Generates 60-80 tokens/sec per user stream.

Tensor Parallelism vs Pipeline Parallelism

⚠️ Good Solution: Pipeline Parallelism (PP): Divides layers across GPUs. High throughput for training, but high latency bubble for interactive single-user chat inference.

⭐ Great Solution: Tensor Parallelism (TP) over NVLink: Divides attention matrices across 8 GPUs within the same node. All 8 GPUs compute in parallel with zero pipeline bubble, delivering ultra-fast < 300ms TTFT.