System Design #6 Amazon Selection Monitoring (SDM Architecture) Billions of Products • Millions of Web Sources • Agentic AI & Entity Resolution

Design Amazon Selection Monitoring

Mission: Make everything anyone wants to buy available on Amazon. An end-to-end distributed system & Agentic AI architecture to discover, mine, classify, and deduplicate billions of products from millions of e-commerce websites worldwide, algorithmically prioritizing and programmatically expanding Amazon's global catalog.

Amazon Scale: 50k Products/Sec99.4% Entity Match • Dual Write/OLAP

Ingestion Velocity

48,500items/s

Billions Processed / Week

Discovery Sources

2.45Mdomains

Worldwide Storefronts & Feeds

Entity Match Precision

99.4%accuracy

Siamese Net + OpenSearch k-NN

Selection Gaps Found

18,240ASINs

High-Demand Programmatic Adds

Dual-Storage Bandwidth

85k WCUwrites

1.2 PB Iceberg OLAP Analytics

Simulate Pipeline:

Source Discovery & Crawling Agentic IR Entity Extraction Taxonomy & Dual Storage Entity Resolution (Siamese & Vector k-NN) Programmatic ASIN Ingestion

Click any component box to view SDM architectural deep dive & code

Monitored E-Commerce Sources

5 Domains

bestbuy.comItems: 480,000 • Status: Healthy

Daily

walmart.comItems: 1,200,000 • Status: Healthy

Daily

target.comItems: 650,000 • Status: Healthy

Daily

zalando.deItems: 820,000 • Status: Healthy

Daily

asos.comItems: 340,000 • Status: Healthy

Weekly

Entity Resolution & Gap Stream

4 Evaluated

Sony WH-1000XM5 SilverSony • Score: 98.6% • Matched ASIN B09XS7JWHH

MATCHED ASIN

Nike Air Zoom Alphafly 3 Neon 10.5Nike • Score: 71.2% • High-Demand ASIN Gap -> Created

NEW ASIN GAP

Anker 737 PowerBank 24000mAhAnker • Score: 97.4% • Matched ASIN B09VPHVT2Z

MATCHED ASIN

Dyson Airwrap Multi-Styler CopperDyson • Score: 99.1% • Matched ASIN B0B1VRM7S2

MATCHED ASIN

Live Pipeline Telemetry Console

[12:00:01]Selection Monitoring System initialized. All discovery fleets, Bedrock extraction agents, and OpenSearch vector clusters online.

SDM Technical Vision & Architectural Deep Dives

Comprehensive guide covering charter execution, entity resolution, dual-storage, and SDM hiring standards.

Software Development Manager (SDM) — Selection Monitoring

Role Charter: Lead the technical vision, architectural execution, and team of engineers to make everything anyone wants to buy available on Amazon worldwide.

Amazon L6/L7 SDM

Key Job Responsibilities & Architectural Alignment

JD #1Architectural Vision & PE Collaboration

Partner with Org Leadership and Principal Engineers to define the 3-year vision for External Source Discovery, moving from fragile static web scraping to resilient, Agentic AI-driven web mining that indexes 2.45M+ storefronts globally.

Delivered via: Source Discovery Registry, Distributed ECS Fargate Fleet, S3 Bronze Lake.

JD #2Scalable Services, Agentic AI & Entity Resolution

Build high-throughput, fault-tolerant microservices and AI Agents solving O(N²) deduplication against 1B+ catalog ASINs using 3-stage candidate blocking, Amazon Bedrock self-healing extractors, and SageMaker Siamese neural matching.

Delivered via: Bedrock Extraction Agents, OpenSearch Vector k-NN, Siamese Cross-Attention.

JD #3Dual-Storage Architecture (Write-Heavy vs. Read-Heavy)

Decouple high-velocity write ingestion from complex downstream analytical reporting. Amazon DynamoDB absorbs 50,000+ writes/s for crawler state and ETags, while DynamoDB Streams / Kinesis continuously feed an Apache Iceberg / Redshift Serverless OLAP lake.

Delivered via: DynamoDB (Write Tier) & Redshift/Iceberg on S3 (Analytical OLAP Tier).

JD #4 & #5High-Performing Team, Mentorship & Operational Excellence

Recruit, mentor, and foster career development for SDE I, II, and Senior SDEs. Establish rigorous standards for automated anomaly detection, regression testing, rollback canary safety, and weekly Operational Excellence (OE) reviews.

Delivered via: Automated SLO Dashboards, 99.4% Precision Guardrails, Canary Deployments.

Amazon Leadership Principles (LPs) Applied to Selection Monitoring

Customer Obsession: If a customer searches for a product on Amazon and can't find it, they leave. Selection Monitoring proactively identifies gaps before customers even search.

Invent and Simplify: Replacing thousands of fragile handcrafted HTML regex scrapers with self-healing Agentic IR models using Amazon Bedrock.

Deliver Results: Converting millions of unorganized external crawl events into prioritized, programmatically added ASINs with quantified GMV impact.

Entity Resolution & Deduplication (Siamese)

Role & Overview

Determines whether an external product is already sold on Amazon or represents an unrepresented selection gap.

Architecture Implementation

1. Stage 1 (Deterministic): GTIN/UPC/EAN exact match (resolves ~45% of items). 2. Stage 2 (Blocking): Constrains search to ASINs in identical category and MinHash LSH buckets. 3. Stage 3 (Neural): SageMaker Siamese transformer calculates cosine similarity across multimodal embeddings.

Scaling Entity Resolution Against 1 Billion ASINs

❌ Bad Solution: Naive Pairwise Comparison: Comparing 50M daily crawled items against 1B catalog items requires 50 quadrillion comparisons (O(N*M))—computationally impossible.

⚠️ Good Solution: Exact String Match: Fast, but fails on slight variations ('Sony WH-1000XM5 Silver' vs 'Sony 1000XM5 Noise-Cancelling Headphones - Platinum Silver').

⭐ Great Solution: Multi-Tier Resolution Pipeline: Deterministic GTIN Match -> Taxonomy Blocking -> OpenSearch Vector k-NN (top-20 candidates) -> Siamese Neural Cross-Attention. Evaluates items in <15ms!

Specification & Snippet

# Siamese Matching Decision Rule
def evaluate_entity_match(candidate_vec, catalog_vecs, threshold=0.94):
    cosine_sim = np.dot(catalog_vecs, candidate_vec) / (np.linalg.norm(catalog_vecs) * np.linalg.norm(candidate_vec))
    best_idx = np.argmax(cosine_sim)
    best_score = cosine_sim[best_idx]
    
    if best_score >= threshold:
        return {"action": "MATCHED_ASIN", "asin": catalog_asins[best_idx], "confidence": best_score}
    else:
        return {"action": "NEW_SELECTION_GAP", "confidence": best_score}