Learnings

System design, part 2 of 12

Building Blocks

Every high-level design answer is assembled from the same two dozen components. This is the reference page for them: what each block is, how it works at interview depth, the options side by side, when to use or avoid it, how it fails, and the numbers worth remembering. All numbers are approximate, order-of-magnitude figures. The method for using these blocks in an answer is in System Design Foundations.

The blocks on one page

A request passes through the edge, reaches stateless compute, reads through a cache, lands in storage, and hands slow work to a queue. Each tier below is one part of this page.

Solid edges are synchronous and sit on the user's latency path. Dashed edges are asynchronous and do not.
PartBlocksThe question it answers
The edgeDNS, CDN, load balancer, reverse proxy, API gatewayHow does a request find a healthy server, cheaply and close to the user?
ComputeStateless services, autoscaling, discovery, poolsHow do we add capacity by adding machines?
CachingLayers, strategies, eviction, failure patternsHow do we avoid doing the same work twice?
Storage choicesDatabase families, object storageWhich store fits this data and this query?
Scaling dataReplication, sharding, consistent hashingWhat happens when one node is not enough?
Asynchronous workQueues, logs, pub-sub, schedulersWhat can leave the request path?
Specialised blocksRate limiter, IDs, Bloom filter, search, geo, mediaWhich named component solves this sub-problem?

The edge

Part 1 of 7 · everything between the user and your code

The edge finds a healthy server for each request, terminates TLS, and serves whatever it can without touching the origin. It is also the first place to shed load and reject bad traffic.

DNS

The edge

DNS maps a name to addresses through a hierarchy of cached lookups. Because every answer is cached for its TTL, DNS is also a slow but global traffic-steering tool.

Only a cold lookup walks the whole chain. The browser, the OS and the resolver each cache the answer for the TTL, so most lookups cost nothing.
RecordMapsWhy it matters in a design
A / AAAAName to IPv4 / IPv6 addressSeveral addresses in one answer is the crudest load balancing
CNAMEName to another nameHow a hostname is pointed at a CDN or a cloud load balancer. Not allowed at the zone apex
ALIAS / ANAMEApex name to another nameProvider feature that works around the apex CNAME rule
NSZone to its authoritative serversDelegation. Use two DNS providers to survive a provider outage
MX, TXT, SRVMail hosts, free text, service host and portTXT carries domain verification and mail policy. SRV is used by some service discovery
  • TTL: long (hours) means fewer lookups and slow change. Short (30 to 60 s) means fast failover and more resolver load. Some resolvers and clients ignore or extend short TTLs, and long-lived connections never re-resolve, so DNS failover is tens of seconds to minutes, never instant.
  • DNS-based load balancing: the authoritative server returns different answers per query: weighted (canary, migration), latency-based or geo (nearest region, data residency), failover (drop an address whose health check fails).
  • Anycast: the same IP address is announced from many locations and the network routes each packet to the nearest one. It is how DNS providers and CDNs get users to a nearby site without DNS tricks.
  • Numbers: cached lookup about 0 to 1 ms, cold lookup roughly 20 to 120 ms.

Failure mode: DNS is a single dependency for everything. Its outages are total and its mistakes are cached for the TTL. Lower the TTL before a planned migration, not during it.

CDN

The edge

A CDN is a fleet of caching reverse proxies in points of presence (PoPs) near users. It cuts latency, takes read traffic and bandwidth off the origin, and absorbs volumetric attacks.

A pull CDN fills itself on the first miss. Every later request in that PoP is served without touching the origin until the TTL expires.
Pull CDNPush CDN
How content arrivesEdge fetches from origin on first missYou upload to the CDN ahead of time
First requestSlow (origin round trip) per PoPFast everywhere
Operational costLow: set cache headersYou manage uploads, expiry and storage
Use whenDefault. Many objects, long tailFew large objects known in advance: releases, video segments, game patches
  • Cache key: by default host + path + query string. Each extra dimension (a header via Vary, a cookie, a device class) multiplies the number of variants and lowers the hit ratio. Normalise query strings and never key on a session cookie.
  • Invalidation, best first: versioned URLs (app.3f9a.js, cached for a year, never purged), short TTL plus stale-while-revalidate, purge API by URL or tag (takes seconds to minutes to reach every PoP).
  • Edge compute: small functions run in the PoP for redirects, header rewrites, token checks and A/B routing.
  • Put on it: static assets, images, video segments, downloads, public API responses that many users share. Keep off it: per-user responses unless the cache key includes the user and they are marked private.
  • Numbers: edge hit 10 to 30 ms versus 100 to 300 ms cross-continent to origin. A healthy static site has a hit ratio above 90%.

Failure mode: a purge or a cold PoP sends the full load to the origin at once. Size the origin for a cache-cold event or put a second cache tier (origin shield) in front of it.

Load balancers: L4 vs L7

The edge

A load balancer spreads connections or requests over a pool of servers and stops sending to the unhealthy ones. The main choice is the layer it works at.

A common large-scale layout: a simple, very fast L4 tier spreads connections over a fleet of L7 proxies, which do the content-aware routing.
L4 (transport)L7 (application)
SeesIP addresses, ports, TCP / UDPHTTP method, path, headers, cookies, gRPC
Unit balancedConnectionRequest
RoutingBy address and port onlyBy host, path, header. Canary and blue-green splits
TLSUsually passed throughTerminated, optionally re-encrypted to the backend
ExtrasNone. Preserves client IP easilyRetries, timeouts, compression, auth, rate limits, caching
Cost per requestVery low. Millions of connections per nodeHigher: parses every request. Tens of thousands of RPS per core
Use whenRaw TCP / UDP, databases, the tier in front of L7HTTP and gRPC services. The default for web APIs
ProductsAWS NLB, LVS, Maglev-styleNGINX, HAProxy, Envoy, AWS ALB

Senior signal: long-lived HTTP/2 and gRPC connections defeat L4 balancing, because one connection carries thousands of requests to a single backend. Balance gRPC at L7 or in the client.

Balancing algorithms

The edge

AlgorithmPicksGood forWeakness
Round robinNext server in turnIdentical servers, short uniform requestsIgnores load. Slow requests pile up on one server
Weighted round robinIn turn, in proportion to a weightMixed instance sizes, canary at 1%Weights are static
Least connectionsServer with fewest open requestsVariable request duration, long-lived connectionsA new or failing-fast server looks idle and gets flooded. Needs slow start
Consistent hashServer owning hash(key)Cache affinity, sharded or stateful backendsUneven load with hot keys. See consistent hashing
Power of two choicesTwo servers at random, then the less loaded of the twoMany balancers with a stale view of loadSlightly worse than a perfect global view
  • Why power of two choices: when many balancers all pick "the least loaded server" from the same stale data, they stampede onto it. Two random candidates removes the herd and gets most of the benefit of full knowledge.
  • Default answer: least connections or power of two choices for request traffic, consistent hash only when a request must reach a particular backend.

Health checks, pairs, stickiness, TLS

The edge

The balancer must not be the single point of failure. The standby takes over the virtual IP when heartbeats stop. Cloud load balancers hide this pair from you.
  • Health checks: active probes (GET /healthz every 2 to 10 s, eject after 2 or 3 failures) and passive checks (eject on real error rate or timeouts, also called outlier detection). Use both. Re-admit gradually.
  • Active-passive pair: simple, and half the capacity is idle. Active-active behind anycast or DNS uses all capacity, and each node must then be able to carry the full load alone.
  • Sticky sessions: pin a user to a server by cookie or source IP. Avoid: load becomes uneven, a server loss drops its sessions, and scale-in and deploys get harder. Keep session state in a shared store or a signed token instead. Acceptable for WebSocket connections, which are pinned by nature.
  • TLS termination: terminate at the balancer to centralise certificates and to let L7 see the request. Re-encrypt to the backend (often mutual TLS) when the internal network is not trusted. A full handshake costs 1 round trip in TLS 1.3 and 2 in TLS 1.2, so keep connections alive and resume sessions.

Failure mode: a health check that tests a shared dependency (the database) fails on every server at once and the balancer ejects the whole pool. Keep the check shallow and fail open when most of the pool looks unhealthy.

Reverse proxy and API gateway

The edge

A reverse proxy is any server that accepts requests on behalf of backends. A load balancer is a reverse proxy whose job is distribution. An API gateway is a reverse proxy whose job is API policy.

Cross-cutting concerns are done once at the gateway so that each service does not re-implement them.
Load balancerReverse proxyAPI gateway
Main jobSpread load, hide failuresFront one or more backendsEnforce API policy per client and per route
Typical featuresAlgorithms, health checksTLS, compression, caching, static filesAuthentication, rate limits and quotas, routing by version, request and response shaping, aggregation
Knows aboutServersSites and pathsAPI keys, users, plans, routes
ProductsNLB, HAProxyNGINX, EnvoyKong, Apigee, AWS API Gateway
  • Request shaping: size limits, header normalisation, protocol translation (REST to gRPC), response trimming for mobile.
  • Use a gateway when many services share auth, quotas and versioning, or third parties call the API. Avoid putting business logic in it: it becomes a shared monolith that every team must deploy through.

Trade-off: the gateway adds a hop (about 1 to 5 ms) and is on the path of every request, so it needs the same redundancy as the load balancer. Details of gateway patterns are in API and Communication Design.

Compute

Part 2 of 7 · servers you can add and remove freely

Compute scales horizontally only if any instance can serve any request. Everything in this part follows from pushing state out of the process.

Stateless services

Compute

A stateless service keeps nothing between requests that another instance would need. Sessions, uploads, locks and counters live in shared stores.

Instances are interchangeable, so any of them can be added, killed or replaced mid-traffic. The state tier is now the part that is hard to scale.
Vertical scaling (bigger machine)Horizontal scaling (more machines)
LimitLargest machine availableCoordination and the shared state tier
FailureOne machine is a single point of failureLoses 1 / N of capacity
ComplexityNoneLoad balancing, discovery, externalised state
Use whenEarly stage, and stateful stores for as long as possibleStateless tiers, always
  • Where session state goes: a signed token in the client (no lookup, hard to revoke) or a session store such as Redis (one lookup, easy to revoke).
  • Capacity rule: run N + 1 or N + 2 so that losing an instance or a zone leaves utilisation below about 70%. A typical request-serving instance handles hundreds to a few thousand RPS, depending on the work per request.

Autoscaling signals

Compute

Scale on the signal that saturates first for the workload, and prefer a signal that leads demand over one that lags it.

SignalFitsWatch out for
CPU utilisation (target 50 to 60%)CPU-bound request handlersUseless for I/O-bound services that stall at 20% CPU
Requests per instanceUniform request costNeeds a known per-instance capacity from load tests
Queue depth or consumer lagWorkers behind a queueScale on backlog divided by processing rate, not raw depth
Latency (p95 / p99)User-facing SLOLagging, and a slow dependency triggers pointless scale-out
Schedule or forecastDaily peaks, launches, salesWrong forecast. Combine with a reactive signal
  • Scale out fast, scale in slowly: use a cooldown and step limits to stop flapping.
  • Reaction time is detection + provisioning + warm-up: tens of seconds for containers, minutes for virtual machines. Keep headroom for the spike that arrives faster than that.

Failure mode: scaling the stateless tier moves the bottleneck to the database, usually as connection exhaustion. Cap the maximum instance count by what the tier behind it can take.

Service discovery

Compute

Instances come and go, so callers need a live list of healthy addresses. Instances register themselves (or the platform registers them) and health checks or heartbeats remove the dead ones.

Client-side: the caller holds the instance list and balances itself. Server-side: the caller knows one stable name and a proxy does the rest.
Client-sideServer-side
Extra hopNoYes (proxy)
Balancing logic lives inA library in every language you useOne proxy tier
Smart routingEasy: per-request choice, zone awarenessWhatever the proxy supports
ExamplesEureka with a client library, gRPC with a resolverKubernetes Service, cloud load balancer, internal DNS
  • Service mesh: a sidecar proxy next to each instance does client-side discovery without a library in the application.
  • Registry: a strongly consistent store such as etcd, ZooKeeper or Consul. Callers must cache the last known list so that a registry outage does not stop traffic.

Connection pooling

Compute

A pool keeps a bounded set of open connections and lends them to requests. It removes per-request connection setup (TCP + TLS + authentication, several round trips) and caps the load on the backend.

300 instances with 10 connections each would open 3,000 database connections. A pooling proxy multiplexes them onto the 100 the database can serve well.
  • Sizing: instances x pool size must stay below the database connection limit, with room for admin and migration sessions. A database does useful work on roughly as many connections as it has cores, so small pools with a short wait queue beat large pools.
  • Why the limit is low: a PostgreSQL connection is a server process costing several MB, so limits are usually in the hundreds. Use a proxy (PgBouncer, RDS Proxy, ProxySQL) once the instance count is large or when using serverless functions.
  • Settings that matter: acquire timeout (fail fast instead of queueing forever), max lifetime (so connections rebalance after a failover), validation on borrow.

Failure mode: one slow query holds connections, the pool empties, every request waits on acquire and the service appears down while the database is merely slow. Set query timeouts shorter than the acquire timeout.

Caching

Part 3 of 7 · trade freshness and memory for latency and load

A cache keeps the result of expensive work close to where it is needed. Every caching decision is three choices: where it lives, how it is filled and written, and how stale data leaves.

Where caches live

Caching

A request stops at the first layer that has the answer. Layers to the left are faster and cheaper, and harder to invalidate.
LayerHoldsLatencyInvalidation
Client (browser, app)Static assets, API responses with Cache-Control or ETag0 msNone once sent: TTL or a new URL only
CDNShared public content10 to 30 msPurge API, seconds to minutes
Gateway / reverse proxyWhole responses for hot endpointsAbout 1 msTTL or purge
In-processSmall hot set: config, feature flags, hot objectsUnder 1 microsecondHard: each instance has its own copy. Short TTL or a pub-sub invalidation message
Distributed (Redis, Memcached)Objects, query results, sessions, counters0.5 to 1 ms in the same zoneExplicit delete or TTL
DatabasePages in the buffer pool, materialised viewsPart of the queryManaged by the database

Interview tip: name the layer and the key when you say "add a cache": "a Redis cache-aside on product:{id}, TTL 5 minutes, deleted on update" is an answer. "We cache it" is not.

Cache-aside

Caching

The application owns the logic: read the cache, on a miss read the database and fill the cache, on a write update the database and delete the key. Also called lazy loading. It is the default.

Only requested data is cached, and a cache outage degrades to slower reads instead of errors.
  • Use when: read-heavy, misses are tolerable, the cached shape differs from the table (joined or computed objects).
  • Pay: every miss costs three round trips, and there is a stale window between the database write and the delete, plus a rare race.
  • Delete, do not set, on write: two concurrent writers that both set can leave the older value in the cache. Delete is idempotent and order-free.

Read-through

Caching

The same read flow as cache-aside, with the loading moved into the cache layer or a caching library. The application only ever talks to the cache.

The loader lives in one place, which makes request coalescing and refresh policies easy to add.
  • Use when: a library or a cache service supports loaders (Caffeine, Guava, a CDN, DynamoDB DAX) and you want uniform behaviour for many callers.
  • Pay: the cached model must match what the loader can fetch, and the cache is now on the critical path for reads.

Write-through

Caching

Every write goes to the cache and the database synchronously, as one operation, before the caller gets a reply. Reads then always hit.

The cache is never behind the database, at the price of a slower write.
  • Use when: data is read soon after it is written and stale reads are not acceptable. Usually paired with read-through.
  • Pay: write latency includes both stores, data that is never read still fills the cache (add a TTL), and a failure between the two writes needs a rule: write the database first and treat the cache write as best effort.

Write-behind

Caching

The write is acknowledged from the cache and flushed to the database later, in batches. Also called write-back.

Many small writes collapse into one database write. Anything not yet flushed is lost if the cache node dies.
  • Use when: write-heavy and loss-tolerant: view counters, likes, last-seen timestamps, metrics.
  • Avoid when: the write is money, inventory or anything that must survive a crash. If you need both speed and durability, write to a durable log first (see queues and logs).

Refresh-ahead

Caching

The cache reloads a hot entry in the background shortly before it expires, so readers never see the miss.

Readers always get an answer from memory. The reload happens off the request path.
  • Use when: a small set of very hot keys is expensive to rebuild: home page, leaderboards, configuration.
  • Pay: wasted reloads for keys nobody asks for again. Refresh only keys read within the last TTL.

Choosing a caching strategy

Caching

StrategyConsistency with the databaseRead latencyWrite latencyIf the cache fails
Cache-asideStale until delete or TTLFast on hit, slow on missDatabase + one deleteReads fall back to the database. Watch the load
Read-throughStale until TTL or invalidationFast on hit, slow on missUnchangedReads fail unless the library falls back
Write-throughFreshAlways fastSlower: two writesWrites fail or bypass, by policy
Write-behindDatabase lags the cacheAlways fastFastestUnflushed writes are lost
Refresh-aheadStale by at most one refresh intervalAlways fast for hot keysUnchangedServes stale, then misses

Senior signal: state the staleness you accept as a number ("product price may be up to 60 s old, stock is never cached") and pick the strategy from that. Do not cache what must be read-after-write correct unless you also say how it is invalidated.

Eviction policies

Caching

Eviction decides what leaves when memory is full. Expiry (TTL) decides what leaves because it is old. A cache needs both.

PolicyEvictsGood forWeakness
LRULeast recently usedGeneral default: recent access predicts the next oneA one-off scan over cold keys flushes the hot set
LFULeast frequently usedStable popularity: catalogue, mediaOld favourites linger unless counts decay. New keys are evicted early
FIFOOldest insertedCheap. Data that ages out naturallyEvicts hot keys just for being old
TTLExpired entries, or soonest to expireBounding stalenessNot a memory policy by itself
  • Choosing: start with LRU plus a TTL on every key. Move to LFU when the hit ratio is poor and popularity is skewed and stable. Modern caches use hybrids (for example W-TinyLFU in Caffeine) that admit a new key only if it is likely to be more useful than the victim.
  • In Redis: eviction is approximate (it samples keys). allkeys-lru or allkeys-lfu for a pure cache. The default noeviction returns errors on writes when memory is full, which is right for a data store and wrong for a cache.

Cache invalidation

Caching

ApproachHowStalenessCost
TTL onlyLet entries expireUp to the TTLNone. Always have it as the safety net
Delete on writeWriter deletes the keys it affectsMillisecondsThe writer must know every key derived from the row
Event-drivenConsumer of the database change stream (CDC) deletes keysSub-second to secondsA pipeline to run. Decouples writers from cache keys
Versioned keysPut a version in the key (user:42:v7) and bump itNone for readers that know the versionAn extra lookup for the version. Old keys expire by TTL
  • The hard cases are derived keys: lists, counts and search pages that depend on many rows. Prefer short TTLs for those and precise invalidation for single-entity keys.

Stampede (thundering herd)

Caching

A hot key expires, and every concurrent request misses and recomputes it at once. The database sees the whole read load for that key in one burst.

With a per-key lock only one request rebuilds the value. The others wait briefly or are given the stale value.
FixHow it worksNote
LockingPer-key mutex in the cache. One rebuilds, the rest wait or serve staleThe lock needs a TTL so a crashed holder does not block the key
Request coalescingIn one process, concurrent callers for a key share one in-flight load ("single flight")Cuts the herd to one per instance, with no extra round trip
Jittered TTLTTL = base + random 0 to 10 or 20%Stops keys that were written together from expiring together
Early refreshRefresh before expiry, either by a background job or probabilistically as the TTL nears its endThe hot key never actually misses

Penetration, avalanche, hot keys, big keys

Caching

ProblemWhat happensFixes
PenetrationRequests for keys that do not exist miss the cache every time and always reach the database. Often an attack or a bugNegative caching (store "not found" with a short TTL), a Bloom filter of valid ids in front, input validation
AvalancheA large share of keys expires together, or the cache cluster goes down, and the database takes the full read loadJittered TTLs, replicas and multi-zone cache, warm-up before taking traffic, load shedding and circuit breakers on the database path
Hot keyOne key takes a large share of traffic and overloads the single shard that owns itIn-process cache in front, read from replicas, split into key#1..N copies and pick one at random
Big keyOne value of many MB, or a collection with millions of members, blocks the server while it is read, deleted or migratedSplit into chunks or buckets, page through collections, delete asynchronously, compress, store the blob elsewhere and cache a pointer
Two defences against penetration: the Bloom filter rejects ids that cannot exist, and negative caching absorbs the rest.

Failure mode: a cache that hides an undersized database. If the system cannot survive a cold cache, the cache is a hard dependency and needs the availability of a database: replicas, failover, and a tested warm-up plan.

Stale data and double delete

Caching

Cache-aside has a race: a reader that missed can write an old value back after a writer has deleted the key. The stale value then lives for a full TTL.

The delayed second delete removes whatever a slow reader put back. The delay should exceed one read round trip plus replication lag.
  • Double delete: delete, then delete again after a few hundred ms to a second, usually from a queue so that it is retried. It narrows the window and does not close it.
  • Also happens with replicas: the reader fills the cache from a lagging replica after the delete. The delay must cover replication lag, or cache fills must read the leader.
  • Stronger fixes: invalidate from the change stream in commit order, or store a version with the value and only accept a set with a newer version (compare-and-set). A TTL remains the backstop.
  • Order of operations: write the database, then delete the cache. Deleting first lets a reader refill the old value before the write commits.

Redis vs Memcached

Caching

RedisMemcached
Data modelStrings, hashes, lists, sets, sorted sets, streams, bitmaps, HyperLogLog, geoOpaque string values only
ThreadingCommands run on one thread (I/O can be threaded), so each command is atomicMulti-threaded: uses all cores of one node
PersistenceOptional snapshots and append-only fileNone
Replication and failoverBuilt in: replicas, Sentinel, ClusterNone built in
ShardingRedis Cluster: 16,384 hash slots assigned to nodesClient-side, usually consistent hashing
ExtrasAtomic scripts, transactions, pub-sub, per-key TTL, many eviction policiesLRU eviction, per-key TTL
Value sizeUp to 512 MB per string (keep values small anyway)1 MB by default
Pick it forAlmost everything: cache plus counters, leaderboards, rate limits, sessions, locks, queuesA plain, very large, multi-threaded key-value cache with simple operations
  • Numbers: either does on the order of 100,000 simple operations per second per node, with sub-millisecond latency inside a zone. Network round trips dominate, so pipeline or batch (MGET).
  • Failure mode: Redis replication is asynchronous, so a failover can lose the last acknowledged writes. Do not treat it as the system of record unless you accept that.

Sizing a cache

Caching

Access is skewed: roughly 20% of the items take 80% of the reads. Cache the hot 20% and you take most of the load off the database for a fifth of the memory.

  • Check throughput too: 20,000 QPS over 3 shards is about 7,000 operations per second per node, far below what a node can do. Memory, not CPU, sets the node count here.
  • Measure, then adjust: the real signal is the hit ratio and the eviction rate. Evictions with a low hit ratio mean the cache is too small or the TTL too long for the key space.
  • TTL from the staleness budget, not from memory: memory pressure is handled by eviction.

Storage choices

Part 4 of 7 · pick the store from the data model and the queries

Choose a database by the shape of the data, the access patterns and the consistency needed, in that order. Most systems use more than one store: a system of record plus derived stores for search, analytics and caching. Storage engines, indexes, transactions and isolation are covered in Data Systems.

Database families

Storage choices

FamilyData modelStrengthsWeaknessesTypical productsPick it when the question is
RelationalTables, rows, foreign keys, SQLACID transactions, joins, ad hoc queries, constraintsWrite scaling beyond one node needs sharding or a distributed SQL enginePostgreSQL, MySQL, Spanner, CockroachDB"Do several records have to change together, correctly?"
Key-valueKey to opaque valueSimplest, fastest, scales by partitioning the keyLookup by key only. No queries on the valueRedis, DynamoDB, Riak"Is every access a get or put by one known key?"
DocumentJSON-like documents, nested fieldsFlexible schema, whole aggregate in one read, secondary indexesJoins and multi-document invariants are weak or costly. DuplicationMongoDB, Couchbase, Firestore"Is the data a self-contained aggregate read as a whole?"
Wide-columnPartition key + sorted clustering columnsVery high write throughput, linear scale, time-ordered rows in a partitionQueries must follow the key design. No joins. Usually eventual consistencyCassandra, HBase, Bigtable, ScyllaDB"Is it huge, write-heavy, and always queried by a known key and range?"
GraphNodes, edges, propertiesMulti-hop traversals, path queriesHard to shard. Poor at bulk aggregatesNeo4j, Neptune, JanusGraph"Is the query about relationships several hops deep?"
Time-seriesTimestamped points per series, with tagsFast append, compression, downsampling, retentionHigh-cardinality tags are expensive. Updates are awkwardInfluxDB, TimescaleDB, Prometheus"Is it append-only measurements queried by time window?"
SearchDocuments in an inverted indexFull-text, relevance ranking, facets, fuzzy matchNear-real-time, not transactional. Not a system of recordElasticsearch, OpenSearch, Solr"Does the user type words and expect ranked results?"
Object / blob storeBucket + key to immutable bytesCheap, very durable, effectively unlimited, HTTP accessNo query, no partial update, tens of ms per requestS3, GCS, Azure Blob"Is it a file or a large payload?"
Data warehouseColumnar tables, SQLScans and aggregates over billions of rowsSeconds per query. Poor at point lookups and frequent small writesBigQuery, Snowflake, Redshift, ClickHouse"Is it analytics over history, not serving a user request?"

Interview tip: justify each store in one sentence from the access pattern: "orders in PostgreSQL because checkout updates order, payment and stock rows in one transaction, product search in Elasticsearch fed by change events, images in object storage behind the CDN".

SQL vs NoSQL, stated carefully

Storage choices

The real difference is the data model and the shape in which each scales, not that one scales and the other does not. Relational databases run at very large scale with replicas and sharding. NoSQL stores scale out more easily because they give up joins and cross-partition transactions, which are exactly the features that are hard to distribute.

DimensionRelationalTypical NoSQL (key-value, document, wide-column)
Design starts fromThe entities: normalise, then query any wayThe queries: one table or collection per access pattern, denormalised
SchemaEnforced on writeFlexible, enforced by the application on read
TransactionsMulti-row ACID is the defaultSingle key or single partition by default. Wider ones are limited or cost more
JoinsIn the databaseIn the application, or avoided by duplication
Scaling shapeScale up, add read replicas, then shard (manual, or built in with distributed SQL)Partitioned from day one: add nodes for capacity
ConsistencyStrong on the leaderOften tunable or eventual
New query laterAdd an indexMay need a new table and a backfill
  • Default to relational when access patterns are still changing or invariants span records. A single well-indexed node handles thousands of writes and tens of thousands of simple reads per second, and terabytes of data.
  • Choose NoSQL when the access pattern is known and simple, the write volume or data size is beyond one node, and you can live without cross-partition joins and transactions.
  • The lines blur: PostgreSQL stores JSON, DynamoDB and MongoDB have transactions, and distributed SQL gives relational semantics on a partitioned store at a latency cost.

Object storage and large files

Storage choices

Never put large binaries in the database or send them through application servers. Store bytes in an object store, keep metadata in a database, and let clients transfer directly with short-lived signed URLs.

The bytes never pass through the application tier. It only issues URLs and records metadata.
  • Metadata vs blob split: the database row holds owner, name, size, checksum, status and the object key. It is small, queryable and transactional. The blob is immutable and addressed by key.
  • Chunking: split a large file into parts (commonly 4 to 64 MB). Parts upload in parallel, a failed part is retried alone, an interrupted upload resumes, and identical chunks can be deduplicated by content hash. The file is then a list of chunk ids in the metadata store.
  • Pre-signed URL: a URL signed by the service that grants one operation on one key for a few minutes. Downloads use the same idea, usually through the CDN.
  • Numbers: S3 is designed for eleven nines of durability. Multipart parts are 5 MB to 5 GB with up to 10,000 parts. First-byte latency is tens of ms, so it is not a low-latency key-value store.
  • Lifecycle: move cold objects to cheaper tiers by age, and delete the parts of abandoned uploads.

Failure mode: orphans. The blob is uploaded and the metadata write fails, or the reverse. Write the metadata row first with a pending status, confirm it on completion, and run a sweeper that removes pending rows and unreferenced blobs.

Scaling data

Part 5 of 7 · more copies, then more pieces

Replication copies the same data to several nodes, for availability and read capacity. Partitioning splits the data over nodes, for write capacity and size. Real systems do both: each shard is a replicated group. Consensus, quorums and consistency models are in Distributed Systems Patterns.

Replication

Scaling data

In leader-follower replication one node accepts all writes and ships its log to followers, which apply it in the same order and serve reads.

One synchronous follower gives a second durable copy of every commit. The asynchronous ones add read capacity and a remote copy without slowing writes.
ModeCommit is acknowledged whenDurability on leader lossWrite latencyIf a follower is down
AsynchronousThe leader has written itRecent writes can be lostLowestNo effect
Synchronous (all)Every follower confirmsNo lossSlowest follower sets itWrites stall
Semi-synchronousAt least one follower confirmsNo loss while one confirmed copy survivesOne extra round tripAnother follower takes the synchronous role, or it degrades to async
  • Use it for: surviving a node loss, serving reads near users, read scaling when reads far outnumber writes. It does not scale writes: every node applies every write.
  • Numbers: same-region lag is usually milliseconds, and grows to seconds or minutes under heavy writes or long transactions on the follower. Cross-region adds the network round trip (50 to 150 ms).
  • Other topologies: multi-leader (writes in several regions, conflicts must be resolved) and leaderless (quorum reads and writes) are in Distributed Systems Patterns.

Replication lag and read-your-writes

Scaling data

With asynchronous followers a read can arrive before the write does. The user saves a profile, reloads, and sees the old one.

The client carries the log position of its last write. A follower may answer only if it has caught up to that position.
AnomalyFixCost
Not seeing your own writeRead the user's own data from the leader, or from the leader for a few seconds after a writeMore leader load
Same, across devicesPosition token as above, stored per userA lookup and occasional fallback
Time going backwards between two readsMonotonic reads: pin each user to one followerLess even load. Re-pin on failure
Stale reads in generalStop routing to followers whose lag exceeds a thresholdLess read capacity when lag is high

Senior signal: say which reads may be stale. "The timeline can lag by seconds and reads from followers. Balance and settings read from the leader."

Failover and split brain

Scaling data

Fencing comes before promotion. Skipping it is how two nodes end up accepting writes.
  • Detection is a guess: a slow node and a dead node look the same. A short timeout causes needless failovers, a long one extends the outage. Typical: 10 to 30 s of missed heartbeats, total failover 30 s to a couple of minutes.
  • Data loss: with async replication the new leader may lack the last writes. When the old leader returns, its extra writes are discarded or need manual repair.
  • Split brain: a network partition leaves two nodes that each believe they are the leader, and both accept writes that then diverge.
  • Prevention: promote only with a majority vote (an odd number of voters, a witness in a third zone), fence the old leader (revoke its lease, cut its storage or network), and attach an increasing epoch number to writes so that storage rejects a stale leader.

Failure mode: automatic failover during an overload. The leader is slow, not dead. Failing over moves the same load onto a colder node and adds a recovery storm. Many teams automate detection and keep a human in the loop for cross-region promotion.

Partitioning and sharding

Scaling data

Sharding splits one logical dataset across nodes so that each holds a part. It is the answer to write throughput or data size beyond one node, and it should come after vertical scaling, read replicas and caching.

Each shard is a replicated group of its own. The router can be a client library, a proxy tier, or the database itself.
SchemeKey to shardGainPayUsed for
RangeContiguous key ranges (A to F, by date)Efficient range scans. Easy to split a rangeHot spots on sequential keys: all new writes hit the last rangeHBase, Bigtable, time-ordered data
Hashhash(key) to a partitionEven spreadNo range scans across keys. Resizing moves data unless done with fixed slots or a ringDynamoDB, Cassandra, Redis Cluster
Directory / lookupA table maps key or tenant to shardFull control: move one tenant, isolate a big oneThe lookup service is a critical dependency and must be cached and replicatedMulti-tenant SaaS, Vitess-style setups
GeoUser's region or countryLow latency, data residencyUneven shard sizes. Cross-region users and queriesRegulated or globally spread user data

Interview tip: give the arithmetic that forces sharding ("2 TB a year and 30,000 writes per second is past one node") before you shard. Sharding without a stated reason is a red flag.

Shard keys, hot partitions, rebalancing

Scaling data

The shard key is the hardest decision to change later. A good key has high cardinality, spreads load evenly, and is present in almost every query so that the query reaches one shard.

Candidate keyResult
user_id for user-owned dataGood: one user's data on one shard, queries are single-shard
created_at or an auto-increment idBad with range partitioning: every write lands on the newest shard
country or statusBad: few values, heavily skewed
tenant_idGood until one tenant outgrows a shard. Use a directory, or a compound key
(user_id, created_at) compoundGood: hashed first part spreads load, sorted second part gives per-user range scans
  • Hot partitions (a celebrity, a flash-sale item): add a suffix to split the key over N sub-partitions and merge on read, give the heavy key its own shard through the directory, or absorb reads with a cache.
  • Rebalancing, never hash mod N: changing N moves almost every key. Use a fixed number of logical partitions much larger than the node count (for example 1,024 or 16,384) and move whole partitions between nodes, or split ranges dynamically when they grow, or use consistent hashing.
  • Moving data live: copy the partition in the background, replicate changes until caught up, switch the routing entry, then delete the old copy. Throttle it: rebalancing competes with production traffic.

Cross-shard queries, joins and transactions

Scaling data

Scatter-gather: latency is that of the slowest shard, and one unavailable shard fails or truncates the whole query.
OperationWhy it gets hardWhat to do instead
Query not on the shard keyMust ask every shard. Tail latency grows with the shard countA global index or a second copy keyed differently, or a search or analytics store
JoinThe rows live on different nodes, so the join moves data over the networkCo-locate related tables on the same key, denormalise, or join in the application
TransactionNo single node can commit it. Needs two-phase commit, which blocks if the coordinator fails, or a sagaChoose the key so that a transaction stays in one shard. Otherwise a saga with compensations
Unique constraint, auto-incrementNo shard sees all the rowsInclude the shard key in the constraint. Generate ids outside the database (see unique IDs)
Sorting and pagingEach shard returns its top N, and deep pages multiply the workCursor paging on the shard key, or a precomputed view

Trade-off: sharding buys write scale and pays with every feature that needs a global view. Two-phase commit and sagas are covered in Distributed Systems Patterns.

Consistent hashing

Scaling data

Consistent hashing maps keys to nodes so that adding or removing a node moves only about 1 / N of the keys, where hash mod N moves nearly all of them.

How it works: hash both nodes and keys onto the same circular space (0 to 232 - 1). A key belongs to the first node found going clockwise from the key's position.

The ring on a 0 to 99 scale. Each key walks clockwise to the next node: 22 goes to B, 60 goes to C, 90 wraps around to A.
EventWhat movesWhat does not
Node D joins at 55Keys in (40, 55] move from C to D. The key at 60 stays on CEverything on A and B
Node B leavesKeys in (10, 40] move from B to C, the next node clockwiseEverything on A, and C's existing keys
Same change with hash mod NAbout N / (N + 1) of all keys change nodeAlmost nothing
  • Problem with one point per node: arcs are uneven, so load is uneven, and a departing node dumps its whole arc onto one neighbour, which can then fail in turn.
  • Virtual nodes: place each physical node at many positions (100 to 200 is common). Arcs average out, a leaving node's keys spread over all the others, and a bigger machine can be given more positions.
  • Replication on the ring: store each key on the next R distinct physical nodes clockwise. This is the preference list in Dynamo-style stores.
  • Where it is used: Dynamo and Cassandra partitioning, client-side sharding of Memcached (ketama), cache-affinity load balancing, CDN request routing.
  • Alternatives: rendezvous hashing (score every node per key and take the highest: simple, no ring), jump consistent hash (no memory, nodes can only be added or removed at the end), and bounded-load variants that cap any node at a factor above the average.

Senior signal: consistent hashing limits how much data moves. It does not fix a hot key, which lands on one node whatever the ring looks like.

Secondary indexes on sharded data

Scaling data

Data is sharded by one key and queried by another (orders sharded by user_id, looked up by restaurant_id). The index can be kept per shard or as a separate partitioned structure.

A local index makes the read fan out to every shard. A global index sends the read to one index partition and makes the write touch two places.
Local (partitioned by document)Global (partitioned by term)
WriteOne shard, atomic with the rowRow shard + index partition: distributed, usually asynchronous
Read by indexed fieldScatter-gather over all shardsOne index partition, then the rows
ConsistencyIndex always matches the rowIndex can lag the row
Use whenThe query also carries the shard key, or is rareThe query is frequent and does not carry the shard key
ExamplesElasticsearch shards, MongoDB, Cassandra secondary index, DynamoDB local secondary indexDynamoDB global secondary index

Asynchronous work

Part 6 of 7 · take work off the request path

A queue or a log between two components decouples them in time: the producer returns at once, the consumer works at its own pace, and a burst becomes a backlog instead of an outage. The price is eventual completion and duplicate handling.

Message queues vs event streams

Asynchronous work

A queue hands each message to one worker and forgets it once acknowledged. A stream is an append-only log that keeps every event for a retention period, and each reader keeps its own position.

Queue: workers compete and a message is gone after its acknowledgement. Stream: the log stays, and independent groups read it at their own offsets.
Message queueEvent stream / log
UnitA task to do onceA fact that happened
OrderingBest effort, or FIFO within a group at lower throughputTotal order within a partition
RetentionUntil acknowledgedBy time or size (days, or forever), whether or not it was read
Consumer modelCompeting consumers, per-message acknowledgement, redelivery on timeoutConsumer groups that commit an offset per partition
ReplayNoYes: rewind the offset. New consumers can read history
Per-message featuresDelay, priority, routing rules, per-message retry and dead-letteringFew: one slow message blocks its partition
ThroughputThousands to tens of thousands per secondHundreds of thousands to millions per second per cluster
ProductsRabbitMQ, Amazon SQS, ActiveMQApache Kafka, Amazon Kinesis, Pulsar
Use forBackground jobs, emails, work distributionEvent sourcing, change data capture, analytics pipelines, many consumers of the same events
  • Why a log is fast: sequential appends to disk, batching, and consumers that read straight from the page cache. This is Kafka's documented design.
  • Payload size: keep messages small (Kafka's default limit is about 1 MB). Put large payloads in object storage and send the key.

Delivery semantics

Asynchronous work

GuaranteeHow it is achievedFailure outcomeUse for
At-most-onceAcknowledge (or commit the offset) before processing. No retriesMessage can be lost, never duplicatedMetrics, logs, anything where a gap is harmless
At-least-onceProcess, then acknowledge. Redeliver on timeoutNever lost, can be delivered twiceThe default. Requires idempotent consumers
Effectively-onceAt-least-once delivery plus deduplication or idempotent writes, or a transaction that covers the result and the offsetDuplicates arrive and have no effectPayments, counters, inventory

Senior signal: "exactly-once delivery" does not exist over an unreliable network. What exists is exactly-once effect: at-least-once delivery plus an idempotency key that the consumer checks. Idempotency keys, the outbox pattern and transactional messaging are in Distributed Systems Patterns.

Consumer groups and partitions

Asynchronous work

A topic is split into partitions. The producer picks the partition from the hash of a key. Within a consumer group each partition is read by exactly one consumer, which is what preserves order per key.

Inside a group the partitions are divided among consumers. Each group sees every message, so groups are how one stream feeds several independent systems.
  • Parallelism is capped by the partition count: a group with more consumers than partitions has idle consumers. Choose the count for peak throughput with headroom. It can be increased later, and doing so changes which partition a key maps to.
  • Rebalance: when a consumer joins or dies, partitions are reassigned and consumption pauses briefly. Messages processed and not yet committed are delivered again to the new owner.
  • Lag (latest offset minus committed offset) is the health metric. Alert on it and autoscale on it.
  • Durability: each partition is replicated (typically 3 copies). A producer that waits for all in-sync replicas trades latency for no loss on a broker failure.

Dead-letter queues, backpressure, ordering keys

Asynchronous work

A message that keeps failing is parked after a bounded number of attempts, so that it cannot block or starve the healthy ones.
  • Dead-letter queue: holds poison messages with their error and attempt count. Alert on its depth, fix the cause, then redrive. Retry with exponential backoff and jitter first. On a stream, publish the failed event to a retry or dead-letter topic and move on, which gives up strict ordering for that key.
  • Backpressure: what happens when producers outrun consumers. Options, in order of preference: scale consumers on lag, bound the queue and reject or slow producers (HTTP 429, blocking send), shed low-priority work, drop the oldest. An unbounded queue only converts overload into unbounded latency and then an out-of-memory failure.
  • Ordering key: messages with the same key (order id, account id) go to the same partition or message group and are processed in order. Different keys run in parallel. Choose the narrowest key that needs ordering: ordering everything globally means one partition and no parallelism.

Failure mode: retries on an overloaded downstream multiply the load. Cap attempts, back off, and stop consuming (circuit breaker) when the dependency is down, letting the backlog wait in the queue.

Pub-sub and fan-out

Asynchronous work

In publish-subscribe the producer publishes to a topic without knowing who listens, and every subscriber gets its own copy.

Topic plus one queue per subscriber: each consumer gets durability, its own retry policy and its own pace. A new subscriber needs no change to the publisher.
Fan-out choiceHowGainPay
On write (push)At publish time, copy the item to every recipient's inbox or queueReads are a single cheap lookupWrite cost grows with the number of recipients. A sender with millions of followers is a storm
On read (pull)Store once. Each reader assembles its view at read timeCheap writesSlow, expensive reads
HybridPush for ordinary senders, pull for the few with huge audiencesBounded write cost and fast readsTwo code paths to merge
  • Products: SNS + SQS, Google Pub/Sub, Kafka topics with one consumer group per subscriber, Redis pub-sub (not durable: a subscriber that is offline misses the message).

Task queues and schedulers

Asynchronous work

A task queue runs background jobs on a worker pool. A scheduler decides when a job becomes due and hands it to the queue.

The scheduler only makes due jobs visible. Execution, retries and scaling belong to the queue and its workers, and the job store records the outcome.
  • Task contract: idempotent (it will run twice some day), with a timeout, bounded retries with backoff, and a visibility timeout longer than the task's worst case so that a second worker does not pick it up mid-run.
  • Scheduler designs: poll a database table for run_at <= now with row locking (simple, good to thousands of jobs per second), delayed messages in the queue, or a sorted set scored by due time. For long delays keep jobs in durable storage and load only the next few minutes into memory.
  • Exactly one scheduler firing: run several scheduler instances for availability and let one lead (leader election), or let all poll and claim jobs with an atomic update.
  • Isolation: separate queues per priority or job class so that a flood of bulk jobs cannot delay password-reset emails.
  • Products: Celery, Sidekiq, SQS with workers, Temporal or Airflow for multi-step workflows.

Specialised blocks

Part 7 of 7 · named components for recurring sub-problems

These come up inside almost every case study. Knowing the standard options lets you name one in a sentence and spend the time on the trade-off.

Rate limiter

Specialised blocks

A rate limiter caps how many requests a client (user, API key, IP address) may make in a period, to protect capacity and enforce fairness and quotas.

Per-client limits belong at the edge, where rejecting is cheapest: an atomic check-and-decrement in Redis, then pass or 429 with Retry-After. A second, local limiter protects each fragile dependency from the service's own traffic.
AlgorithmHowBurstsMemory per keyAccuracy and notes
Token bucketTokens refill at rate r up to capacity b. A request spends oneAllowed, up to b2 values: tokens, last refill timeThe usual default. Two knobs: sustained rate and burst size
Leaky bucketRequests enter a queue that drains at a fixed rate. Overflow is droppedSmoothed awayQueue or 2 valuesConstant output rate. Adds delay. Good in front of a fragile backend
Fixed windowOne counter per window (per minute)Up to 2x the limit across a window boundary1 counterSimplest. Fine for coarse quotas
Sliding window logStore the timestamp of every request, count those inside the windowNone beyond the limitOne entry per request, up to the limitExact and expensive
Sliding window counterCurrent window count + previous window count weighted by overlapSmall error2 countersApproximate, cheap. Good default for strict limits at scale
  • Distributed counting: keep counters in Redis and make check-and-update atomic (a script or a single increment with expiry). For very high rates, each node takes a local batch of tokens and syncs periodically, trading exactness for no round trip.
  • Layers: edge or gateway for per-client limits, the service for per-tenant business quotas, the client library for polite backoff.
  • When the limiter's store is down: fail open for availability (most APIs), fail closed for abuse-sensitive paths such as login or one-time-password sending.

Interview tip: state the key, the limit and the response: "100 requests per minute per API key, token bucket with burst 20, 429 with Retry-After, limits returned in response headers".

Unique ID generation

Specialised blocks

SchemeSizeSortable by timeCoordinationGainPay
Database auto-increment64 bitsYesThe one databaseSimple, compact, denseSingle writer. Breaks with sharding. Exposes volume and is guessable
UUID v4128 bitsNoNoneGenerate anywhere, no collisions in practice (122 random bits)Large. Random inserts scatter across a B-tree index and hurt write locality
UUID v7128 bitsYes (ms)NoneTimestamp prefix keeps inserts at the end of the indexLarge. Reveals creation time
Snowflake-style64 bitsYes (ms)A machine id per generatorCompact, ordered, generated locally at very high ratesDepends on clocks. Machine ids must be assigned
Ticket server64 bitsRoughlyA dedicated id serviceSimple numeric ids across shardsA network call per id or per block, and a service to keep highly available

The Snowflake layout, as published by Twitter. Variants move the bit boundaries.

FieldBitsHoldsCapacity
Sign1Always 0, so the id is positive in signed types
Timestamp41Milliseconds since a custom epochAbout 69 years
Machine id10Datacenter + worker1,024 generators
Sequence12Counter within the same millisecond4,096 ids per ms per generator
  • Clock moving backwards could repeat an id. The generator must refuse or wait until the clock passes the last timestamp it used.
  • Ticket server: a database that only hands out auto-increment values. Run two with different offsets (one odd, one even) for availability, and hand out blocks of ids (1,000 at a time) to cut round trips. The ids are then only roughly ordered.
  • Choosing: one database: auto-increment. Sharded or generated by many services, with ordering needed: Snowflake-style or UUID v7. Ids created offline or on clients: UUID. Public ids should not be guessable, so do not expose a raw sequence.

Bloom filter and friends

Specialised blocks

A Bloom filter is a bit array with k hash functions that answers "is this key in the set?" with no false negatives: "no" is certain, "yes" means probably.

Adding a key sets k bits. A lookup checks the same k bits: one zero bit proves the key was never added.
  • Sizing intuition: about 10 bits per key with 7 hash functions gives a 1% false-positive rate, whatever the key size. Each further 10x reduction costs about 4.8 more bits per key. One billion keys at 1% is about 1.2 GB.
  • Limits: no deletes (a counting variant allows them at several times the memory), no listing, and the rate rises once more keys are added than it was sized for.
  • Uses: skipping disk reads for absent keys in LSM-tree stores, blocking cache penetration, "already seen" checks for crawled URLs or recommended items, username-taken pre-checks.
  • HyperLogLog estimates the number of distinct items (unique visitors) to within about 1% in roughly 12 KB. Count-Min Sketch estimates how often each item occurred (top-K, heavy hitters) in fixed memory, and can only overestimate.

Search index

Specialised blocks

An inverted index maps each term to the sorted list of documents that contain it (a posting list), so a text query becomes a few list lookups and an intersection instead of a scan.

Documents and queries pass through the same analyzer. The posting lists for "red" and "shoe" intersect at document 1.
  • Ranking: a relevance score such as BM25 (term frequency, rarity of the term, document length), usually blended with business signals such as popularity and recency.
  • Writes: new documents go to small immutable segments that are merged in the background, so search is near-real-time (about a second behind), not transactional.
  • Scaling: the index is split into shards. A query is scatter-gather over all shards, each returning its top N.
  • Use when users search text, need facets or fuzzy matching. Avoid as the system of record: feed it from the primary database through change events and be able to rebuild it.

Geospatial indexing

Specialised blocks

"Find things near this point" is a two-dimensional query, and ordinary indexes are one-dimensional. Every scheme maps an area to a cell id that an ordinary index can store.

The neighbours matter: two points can be metres apart and still fall in different cells.
GeohashQuadtreeS2 / H3
IdeaInterleave latitude and longitude bits into a base-32 string. A shared prefix means the same cellTree that splits a square into 4 whenever it holds too many pointsS2: square cells on a sphere ordered along a space-filling curve, 64-bit ids. H3: hexagonal cells
Cell sizeFixed per length: 5 chars about 5 km, 6 about 1 km, 7 about 150 mAdapts to density: small cells downtown, large in the countrysideFixed per level, many levels
StorageAny database or Redis: a string or integer column with a normal indexUsually an in-memory structure you build and updateAn integer column, with a library to compute cells and coverings
GainSimplest. Prefix queries. Built into Redis and ElasticsearchEven load per leaf. Natural "k nearest" searchLittle distortion. S2 covers arbitrary regions well. H3 neighbours are all at the same distance
PayCell edges, and cells shrink toward the poles. Dense areas overload a cellRebuilding and rebalancing as points move. Harder to shardLibrary dependency, more concepts
Pick forNearby search with a fixed radiusVery uneven density, static or slowly changing pointsRide-hailing and maps at scale, region coverage, aggregation by area
  • Moving objects (drivers): keep the latest position in memory keyed by cell with a short TTL, updated every few seconds. Do not write each update to the durable store.

Blob and media processing pipeline

Specialised blocks

Uploads are processed asynchronously: the original lands in object storage, an event starts a set of workers, and the derived files are published through the CDN.

The upload (a pre-signed PUT) returns as soon as the original is stored. Every later step is a retryable job, and the metadata row moves from processing to ready.
  • Work as a graph: split a video into segments, transcode each segment into several resolutions and bitrates in parallel, then write the manifests. Images get resized variants and a stripped copy without location metadata.
  • Each step is idempotent and writes to a deterministic output key, so a retry overwrites instead of duplicating.
  • Status: the client polls the metadata service or receives a webhook or push when the state becomes ready.
  • Scaling: autoscale workers on queue lag. Use separate queues for short and long jobs so that a 4-hour video does not delay profile pictures.
  • Numbers: transcoding is CPU or GPU bound and takes a meaningful fraction of the video's duration per rendition. It is the cost centre of the pipeline.

Heartbeats, health checks, locks and leaders

Specialised blocks

A heartbeat is a periodic "I am alive" signal. A node that misses several in a row is treated as dead, which is always a guess: the node may only be slow or cut off.

Detection time is the interval times the number of misses allowed. Shorter means faster failover and more false alarms.
CheckAsksOn failureShould test
LivenessIs the process stuck?Restart itOnly the process itself. Never dependencies
ReadinessCan it serve traffic now?Take it out of rotation, do not restartWarm-up done, critical local resources available
Deep / dependencyAre the things it calls healthy?Alert, degrade featuresUsed for dashboards, not for ejecting instances
  • Push vs pull: nodes send heartbeats to a registry (works across network boundaries, the registry must expire entries), or a checker probes the nodes (no client code, the checker must reach everyone).
  • At scale: gossip protocols spread liveness information between peers without a central monitor.
  • Distributed lock: a lease with an expiry and a fencing token, so that only one holder acts at a time. See Distributed Systems Patterns.
  • Leader election: one node from a group is chosen through a consensus-backed store (etcd, ZooKeeper) and keeps the role by renewing a lease. See Distributed Systems Patterns.

Cheat sheet

NeedBlockKey trade-off
Route users to the nearest healthy regionGeo or latency DNS, anycastFailover is only as fast as the TTL
Serve static or shared content fastCDNInvalidation delay. Use versioned URLs
Spread requests, hide dead serversL7 load balancerExtra hop, and it must itself be redundant
Auth, quotas, routing in one placeAPI gatewayCentral choke point. Keep logic out of it
Handle more request trafficStateless services + autoscalingState moves to shared stores, which become the bottleneck
Too many database connectionsConnection pool or pooling proxyPool exhaustion when queries are slow
Cut read latency and database loadCache-aside on RedisStaleness, and survival of a cold cache
Absorb very frequent small writesWrite-behindLoses unflushed writes on a crash
Hot key expiring under loadLock, coalesce, jitter, early refreshWaiters see extra latency or stale data
Lookups for keys that do not existNegative cache, Bloom filterFalse positives, and no deletes
Transactions and flexible queriesRelational databaseWrite scaling needs sharding
Huge write volume by known keyWide-column or key-value storeQueries are fixed by the key design
Files, images, videoObject store + pre-signed URLs + CDNNo queries. Metadata lives elsewhere
Survive a node loss, scale readsLeader-follower replicationLag, and data loss on async failover
Scale writes or data sizeSharding on a well-chosen keyCross-shard joins, transactions and queries
Add or remove nodes with little data movementConsistent hashing with virtual nodesDoes not help with hot keys
Query sharded data by another fieldGlobal secondary indexIndex lags the row. Extra write cost
Run slow work laterMessage queue + workersAt-least-once: handlers must be idempotent
Many consumers, replay, ordered eventsEvent log with consumer groupsOrder only within a partition. Parallelism capped by partitions
Notify several systems of one eventPub-sub, one queue per subscriberFan-out on write grows with the audience
Messages that always failRetries with backoff, dead-letter queueSomeone has to watch and redrive it
Protect an API from abuse or overloadToken bucket at the gatewayShared counter store. Fail open or closed
IDs across shards, time-orderedSnowflake-style or UUID v7Clock dependence, or 128-bit keys
Full-text searchInverted indexNear-real-time, a derived store to keep in sync
Find things nearbyGeohash, quadtree, S2 or H3Cell-edge handling, uneven density
Process uploadsObject store event, queue, workersAsynchronous: the client must track status
Know which nodes are aliveHeartbeats and health checksDetection speed versus false positives

Numbers to keep in your head, all approximate:

ThingOrder of magnitude
Redis or Memcached operation, same zone0.5 to 1 ms, about 100,000 operations per second per node
CDN edge hit vs cross-continent origin10 to 30 ms vs 100 to 300 ms
Object store first byteTens of ms
Replication lag, same region / cross regionMilliseconds normally / 50 to 150 ms and up
Database failover30 s to a couple of minutes
Consistent hashing, node added to NAbout 1 / (N + 1) of keys move
Bloom filter10 bits per key for 1% false positives
Snowflake id41 + 10 + 12 bits: 69 years, 1,024 generators, 4,096 ids per ms each
Kafka message limit by defaultAbout 1 MB

Next: how these stores work inside, from storage engines and indexes to transactions and isolation, in Data Systems.