System design, part 2 of 12
Building Blocks
Every high-level design answer is assembled from the same two dozen components. This is the reference page for them: what each block is, how it works at interview depth, the options side by side, when to use or avoid it, how it fails, and the numbers worth remembering. All numbers are approximate, order-of-magnitude figures. The method for using these blocks in an answer is in System Design Foundations.
The blocks on one page
A request passes through the edge, reaches stateless compute, reads through a cache, lands in storage, and hands slow work to a queue. Each tier below is one part of this page.
| Part | Blocks | The question it answers |
|---|---|---|
| The edge | DNS, CDN, load balancer, reverse proxy, API gateway | How does a request find a healthy server, cheaply and close to the user? |
| Compute | Stateless services, autoscaling, discovery, pools | How do we add capacity by adding machines? |
| Caching | Layers, strategies, eviction, failure patterns | How do we avoid doing the same work twice? |
| Storage choices | Database families, object storage | Which store fits this data and this query? |
| Scaling data | Replication, sharding, consistent hashing | What happens when one node is not enough? |
| Asynchronous work | Queues, logs, pub-sub, schedulers | What can leave the request path? |
| Specialised blocks | Rate limiter, IDs, Bloom filter, search, geo, media | Which named component solves this sub-problem? |
The edge
Part 1 of 7 · everything between the user and your code
The edge finds a healthy server for each request, terminates TLS, and serves whatever it can without touching the origin. It is also the first place to shed load and reject bad traffic.
- DNS: name to address, plus coarse traffic steering
- CDN: cached content served near the user
- Load balancers: L4 vs L7
- Balancing algorithms: round robin to power of two choices
- Health checks, pairs, stickiness, TLS
- Reverse proxy and API gateway
DNS
DNS maps a name to addresses through a hierarchy of cached lookups. Because every answer is cached for its TTL, DNS is also a slow but global traffic-steering tool.
| Record | Maps | Why it matters in a design |
|---|---|---|
| A / AAAA | Name to IPv4 / IPv6 address | Several addresses in one answer is the crudest load balancing |
| CNAME | Name to another name | How a hostname is pointed at a CDN or a cloud load balancer. Not allowed at the zone apex |
| ALIAS / ANAME | Apex name to another name | Provider feature that works around the apex CNAME rule |
| NS | Zone to its authoritative servers | Delegation. Use two DNS providers to survive a provider outage |
| MX, TXT, SRV | Mail hosts, free text, service host and port | TXT carries domain verification and mail policy. SRV is used by some service discovery |
- TTL: long (hours) means fewer lookups and slow change. Short (30 to 60 s) means fast failover and more resolver load. Some resolvers and clients ignore or extend short TTLs, and long-lived connections never re-resolve, so DNS failover is tens of seconds to minutes, never instant.
- DNS-based load balancing: the authoritative server returns different answers per query: weighted (canary, migration), latency-based or geo (nearest region, data residency), failover (drop an address whose health check fails).
- Anycast: the same IP address is announced from many locations and the network routes each packet to the nearest one. It is how DNS providers and CDNs get users to a nearby site without DNS tricks.
- Numbers: cached lookup about 0 to 1 ms, cold lookup roughly 20 to 120 ms.
Failure mode: DNS is a single dependency for everything. Its outages are total and its mistakes are cached for the TTL. Lower the TTL before a planned migration, not during it.
CDN
A CDN is a fleet of caching reverse proxies in points of presence (PoPs) near users. It cuts latency, takes read traffic and bandwidth off the origin, and absorbs volumetric attacks.
| Pull CDN | Push CDN | |
|---|---|---|
| How content arrives | Edge fetches from origin on first miss | You upload to the CDN ahead of time |
| First request | Slow (origin round trip) per PoP | Fast everywhere |
| Operational cost | Low: set cache headers | You manage uploads, expiry and storage |
| Use when | Default. Many objects, long tail | Few large objects known in advance: releases, video segments, game patches |
- Cache key: by default host + path + query string. Each extra dimension (a header via
Vary, a cookie, a device class) multiplies the number of variants and lowers the hit ratio. Normalise query strings and never key on a session cookie. - Invalidation, best first: versioned URLs (
app.3f9a.js, cached for a year, never purged), short TTL plusstale-while-revalidate, purge API by URL or tag (takes seconds to minutes to reach every PoP). - Edge compute: small functions run in the PoP for redirects, header rewrites, token checks and A/B routing.
- Put on it: static assets, images, video segments, downloads, public API responses that many users share. Keep off it: per-user responses unless the cache key includes the user and they are marked
private. - Numbers: edge hit 10 to 30 ms versus 100 to 300 ms cross-continent to origin. A healthy static site has a hit ratio above 90%.
Failure mode: a purge or a cold PoP sends the full load to the origin at once. Size the origin for a cache-cold event or put a second cache tier (origin shield) in front of it.
Load balancers: L4 vs L7
A load balancer spreads connections or requests over a pool of servers and stops sending to the unhealthy ones. The main choice is the layer it works at.
| L4 (transport) | L7 (application) | |
|---|---|---|
| Sees | IP addresses, ports, TCP / UDP | HTTP method, path, headers, cookies, gRPC |
| Unit balanced | Connection | Request |
| Routing | By address and port only | By host, path, header. Canary and blue-green splits |
| TLS | Usually passed through | Terminated, optionally re-encrypted to the backend |
| Extras | None. Preserves client IP easily | Retries, timeouts, compression, auth, rate limits, caching |
| Cost per request | Very low. Millions of connections per node | Higher: parses every request. Tens of thousands of RPS per core |
| Use when | Raw TCP / UDP, databases, the tier in front of L7 | HTTP and gRPC services. The default for web APIs |
| Products | AWS NLB, LVS, Maglev-style | NGINX, HAProxy, Envoy, AWS ALB |
Senior signal: long-lived HTTP/2 and gRPC connections defeat L4 balancing, because one connection carries thousands of requests to a single backend. Balance gRPC at L7 or in the client.
Balancing algorithms
| Algorithm | Picks | Good for | Weakness |
|---|---|---|---|
| Round robin | Next server in turn | Identical servers, short uniform requests | Ignores load. Slow requests pile up on one server |
| Weighted round robin | In turn, in proportion to a weight | Mixed instance sizes, canary at 1% | Weights are static |
| Least connections | Server with fewest open requests | Variable request duration, long-lived connections | A new or failing-fast server looks idle and gets flooded. Needs slow start |
| Consistent hash | Server owning hash(key) | Cache affinity, sharded or stateful backends | Uneven load with hot keys. See consistent hashing |
| Power of two choices | Two servers at random, then the less loaded of the two | Many balancers with a stale view of load | Slightly worse than a perfect global view |
- Why power of two choices: when many balancers all pick "the least loaded server" from the same stale data, they stampede onto it. Two random candidates removes the herd and gets most of the benefit of full knowledge.
- Default answer: least connections or power of two choices for request traffic, consistent hash only when a request must reach a particular backend.
Health checks, pairs, stickiness, TLS
- Health checks: active probes (GET
/healthzevery 2 to 10 s, eject after 2 or 3 failures) and passive checks (eject on real error rate or timeouts, also called outlier detection). Use both. Re-admit gradually. - Active-passive pair: simple, and half the capacity is idle. Active-active behind anycast or DNS uses all capacity, and each node must then be able to carry the full load alone.
- Sticky sessions: pin a user to a server by cookie or source IP. Avoid: load becomes uneven, a server loss drops its sessions, and scale-in and deploys get harder. Keep session state in a shared store or a signed token instead. Acceptable for WebSocket connections, which are pinned by nature.
- TLS termination: terminate at the balancer to centralise certificates and to let L7 see the request. Re-encrypt to the backend (often mutual TLS) when the internal network is not trusted. A full handshake costs 1 round trip in TLS 1.3 and 2 in TLS 1.2, so keep connections alive and resume sessions.
Failure mode: a health check that tests a shared dependency (the database) fails on every server at once and the balancer ejects the whole pool. Keep the check shallow and fail open when most of the pool looks unhealthy.
Reverse proxy and API gateway
A reverse proxy is any server that accepts requests on behalf of backends. A load balancer is a reverse proxy whose job is distribution. An API gateway is a reverse proxy whose job is API policy.
| Load balancer | Reverse proxy | API gateway | |
|---|---|---|---|
| Main job | Spread load, hide failures | Front one or more backends | Enforce API policy per client and per route |
| Typical features | Algorithms, health checks | TLS, compression, caching, static files | Authentication, rate limits and quotas, routing by version, request and response shaping, aggregation |
| Knows about | Servers | Sites and paths | API keys, users, plans, routes |
| Products | NLB, HAProxy | NGINX, Envoy | Kong, Apigee, AWS API Gateway |
- Request shaping: size limits, header normalisation, protocol translation (REST to gRPC), response trimming for mobile.
- Use a gateway when many services share auth, quotas and versioning, or third parties call the API. Avoid putting business logic in it: it becomes a shared monolith that every team must deploy through.
Trade-off: the gateway adds a hop (about 1 to 5 ms) and is on the path of every request, so it needs the same redundancy as the load balancer. Details of gateway patterns are in API and Communication Design.
Compute
Part 2 of 7 · servers you can add and remove freely
Compute scales horizontally only if any instance can serve any request. Everything in this part follows from pushing state out of the process.
- Stateless services and horizontal scaling
- Autoscaling signals
- Service discovery: client-side vs server-side
- Connection pooling
Stateless services
A stateless service keeps nothing between requests that another instance would need. Sessions, uploads, locks and counters live in shared stores.
| Vertical scaling (bigger machine) | Horizontal scaling (more machines) | |
|---|---|---|
| Limit | Largest machine available | Coordination and the shared state tier |
| Failure | One machine is a single point of failure | Loses 1 / N of capacity |
| Complexity | None | Load balancing, discovery, externalised state |
| Use when | Early stage, and stateful stores for as long as possible | Stateless tiers, always |
- Where session state goes: a signed token in the client (no lookup, hard to revoke) or a session store such as Redis (one lookup, easy to revoke).
- Capacity rule: run N + 1 or N + 2 so that losing an instance or a zone leaves utilisation below about 70%. A typical request-serving instance handles hundreds to a few thousand RPS, depending on the work per request.
Autoscaling signals
Scale on the signal that saturates first for the workload, and prefer a signal that leads demand over one that lags it.
| Signal | Fits | Watch out for |
|---|---|---|
| CPU utilisation (target 50 to 60%) | CPU-bound request handlers | Useless for I/O-bound services that stall at 20% CPU |
| Requests per instance | Uniform request cost | Needs a known per-instance capacity from load tests |
| Queue depth or consumer lag | Workers behind a queue | Scale on backlog divided by processing rate, not raw depth |
| Latency (p95 / p99) | User-facing SLO | Lagging, and a slow dependency triggers pointless scale-out |
| Schedule or forecast | Daily peaks, launches, sales | Wrong forecast. Combine with a reactive signal |
- Scale out fast, scale in slowly: use a cooldown and step limits to stop flapping.
- Reaction time is detection + provisioning + warm-up: tens of seconds for containers, minutes for virtual machines. Keep headroom for the spike that arrives faster than that.
Failure mode: scaling the stateless tier moves the bottleneck to the database, usually as connection exhaustion. Cap the maximum instance count by what the tier behind it can take.
Service discovery
Instances come and go, so callers need a live list of healthy addresses. Instances register themselves (or the platform registers them) and health checks or heartbeats remove the dead ones.
| Client-side | Server-side | |
|---|---|---|
| Extra hop | No | Yes (proxy) |
| Balancing logic lives in | A library in every language you use | One proxy tier |
| Smart routing | Easy: per-request choice, zone awareness | Whatever the proxy supports |
| Examples | Eureka with a client library, gRPC with a resolver | Kubernetes Service, cloud load balancer, internal DNS |
- Service mesh: a sidecar proxy next to each instance does client-side discovery without a library in the application.
- Registry: a strongly consistent store such as etcd, ZooKeeper or Consul. Callers must cache the last known list so that a registry outage does not stop traffic.
Connection pooling
A pool keeps a bounded set of open connections and lends them to requests. It removes per-request connection setup (TCP + TLS + authentication, several round trips) and caps the load on the backend.
- Sizing: instances x pool size must stay below the database connection limit, with room for admin and migration sessions. A database does useful work on roughly as many connections as it has cores, so small pools with a short wait queue beat large pools.
- Why the limit is low: a PostgreSQL connection is a server process costing several MB, so limits are usually in the hundreds. Use a proxy (PgBouncer, RDS Proxy, ProxySQL) once the instance count is large or when using serverless functions.
- Settings that matter: acquire timeout (fail fast instead of queueing forever), max lifetime (so connections rebalance after a failover), validation on borrow.
Failure mode: one slow query holds connections, the pool empties, every request waits on acquire and the service appears down while the database is merely slow. Set query timeouts shorter than the acquire timeout.
Caching
Part 3 of 7 · trade freshness and memory for latency and load
A cache keeps the result of expensive work close to where it is needed. Every caching decision is three choices: where it lives, how it is filled and written, and how stale data leaves.
- Where caches live
- Strategies: cache-aside, read-through, write-through, write-behind, refresh-ahead, and how to choose
- Eviction policies and invalidation
- Classic problems: stampede, penetration, avalanche, hot keys, big keys, stale data
- Redis vs Memcached and sizing a cache
Where caches live
| Layer | Holds | Latency | Invalidation |
|---|---|---|---|
| Client (browser, app) | Static assets, API responses with Cache-Control or ETag | 0 ms | None once sent: TTL or a new URL only |
| CDN | Shared public content | 10 to 30 ms | Purge API, seconds to minutes |
| Gateway / reverse proxy | Whole responses for hot endpoints | About 1 ms | TTL or purge |
| In-process | Small hot set: config, feature flags, hot objects | Under 1 microsecond | Hard: each instance has its own copy. Short TTL or a pub-sub invalidation message |
| Distributed (Redis, Memcached) | Objects, query results, sessions, counters | 0.5 to 1 ms in the same zone | Explicit delete or TTL |
| Database | Pages in the buffer pool, materialised views | Part of the query | Managed by the database |
Interview tip: name the layer and the key when you say "add a cache": "a Redis cache-aside on product:{id}, TTL 5 minutes, deleted on update" is an answer. "We cache it" is not.
Cache-aside
The application owns the logic: read the cache, on a miss read the database and fill the cache, on a write update the database and delete the key. Also called lazy loading. It is the default.
- Use when: read-heavy, misses are tolerable, the cached shape differs from the table (joined or computed objects).
- Pay: every miss costs three round trips, and there is a stale window between the database write and the delete, plus a rare race.
- Delete, do not set, on write: two concurrent writers that both set can leave the older value in the cache. Delete is idempotent and order-free.
Read-through
The same read flow as cache-aside, with the loading moved into the cache layer or a caching library. The application only ever talks to the cache.
- Use when: a library or a cache service supports loaders (Caffeine, Guava, a CDN, DynamoDB DAX) and you want uniform behaviour for many callers.
- Pay: the cached model must match what the loader can fetch, and the cache is now on the critical path for reads.
Write-through
Every write goes to the cache and the database synchronously, as one operation, before the caller gets a reply. Reads then always hit.
- Use when: data is read soon after it is written and stale reads are not acceptable. Usually paired with read-through.
- Pay: write latency includes both stores, data that is never read still fills the cache (add a TTL), and a failure between the two writes needs a rule: write the database first and treat the cache write as best effort.
Write-behind
The write is acknowledged from the cache and flushed to the database later, in batches. Also called write-back.
- Use when: write-heavy and loss-tolerant: view counters, likes, last-seen timestamps, metrics.
- Avoid when: the write is money, inventory or anything that must survive a crash. If you need both speed and durability, write to a durable log first (see queues and logs).
Refresh-ahead
The cache reloads a hot entry in the background shortly before it expires, so readers never see the miss.
- Use when: a small set of very hot keys is expensive to rebuild: home page, leaderboards, configuration.
- Pay: wasted reloads for keys nobody asks for again. Refresh only keys read within the last TTL.
Choosing a caching strategy
| Strategy | Consistency with the database | Read latency | Write latency | If the cache fails |
|---|---|---|---|---|
| Cache-aside | Stale until delete or TTL | Fast on hit, slow on miss | Database + one delete | Reads fall back to the database. Watch the load |
| Read-through | Stale until TTL or invalidation | Fast on hit, slow on miss | Unchanged | Reads fail unless the library falls back |
| Write-through | Fresh | Always fast | Slower: two writes | Writes fail or bypass, by policy |
| Write-behind | Database lags the cache | Always fast | Fastest | Unflushed writes are lost |
| Refresh-ahead | Stale by at most one refresh interval | Always fast for hot keys | Unchanged | Serves stale, then misses |
Senior signal: state the staleness you accept as a number ("product price may be up to 60 s old, stock is never cached") and pick the strategy from that. Do not cache what must be read-after-write correct unless you also say how it is invalidated.
Eviction policies
Eviction decides what leaves when memory is full. Expiry (TTL) decides what leaves because it is old. A cache needs both.
| Policy | Evicts | Good for | Weakness |
|---|---|---|---|
| LRU | Least recently used | General default: recent access predicts the next one | A one-off scan over cold keys flushes the hot set |
| LFU | Least frequently used | Stable popularity: catalogue, media | Old favourites linger unless counts decay. New keys are evicted early |
| FIFO | Oldest inserted | Cheap. Data that ages out naturally | Evicts hot keys just for being old |
| TTL | Expired entries, or soonest to expire | Bounding staleness | Not a memory policy by itself |
- Choosing: start with LRU plus a TTL on every key. Move to LFU when the hit ratio is poor and popularity is skewed and stable. Modern caches use hybrids (for example W-TinyLFU in Caffeine) that admit a new key only if it is likely to be more useful than the victim.
- In Redis: eviction is approximate (it samples keys).
allkeys-lruorallkeys-lfufor a pure cache. The defaultnoevictionreturns errors on writes when memory is full, which is right for a data store and wrong for a cache.
Cache invalidation
| Approach | How | Staleness | Cost |
|---|---|---|---|
| TTL only | Let entries expire | Up to the TTL | None. Always have it as the safety net |
| Delete on write | Writer deletes the keys it affects | Milliseconds | The writer must know every key derived from the row |
| Event-driven | Consumer of the database change stream (CDC) deletes keys | Sub-second to seconds | A pipeline to run. Decouples writers from cache keys |
| Versioned keys | Put a version in the key (user:42:v7) and bump it | None for readers that know the version | An extra lookup for the version. Old keys expire by TTL |
- The hard cases are derived keys: lists, counts and search pages that depend on many rows. Prefer short TTLs for those and precise invalidation for single-entity keys.
Stampede (thundering herd)
A hot key expires, and every concurrent request misses and recomputes it at once. The database sees the whole read load for that key in one burst.
| Fix | How it works | Note |
|---|---|---|
| Locking | Per-key mutex in the cache. One rebuilds, the rest wait or serve stale | The lock needs a TTL so a crashed holder does not block the key |
| Request coalescing | In one process, concurrent callers for a key share one in-flight load ("single flight") | Cuts the herd to one per instance, with no extra round trip |
| Jittered TTL | TTL = base + random 0 to 10 or 20% | Stops keys that were written together from expiring together |
| Early refresh | Refresh before expiry, either by a background job or probabilistically as the TTL nears its end | The hot key never actually misses |
Penetration, avalanche, hot keys, big keys
| Problem | What happens | Fixes |
|---|---|---|
| Penetration | Requests for keys that do not exist miss the cache every time and always reach the database. Often an attack or a bug | Negative caching (store "not found" with a short TTL), a Bloom filter of valid ids in front, input validation |
| Avalanche | A large share of keys expires together, or the cache cluster goes down, and the database takes the full read load | Jittered TTLs, replicas and multi-zone cache, warm-up before taking traffic, load shedding and circuit breakers on the database path |
| Hot key | One key takes a large share of traffic and overloads the single shard that owns it | In-process cache in front, read from replicas, split into key#1..N copies and pick one at random |
| Big key | One value of many MB, or a collection with millions of members, blocks the server while it is read, deleted or migrated | Split into chunks or buckets, page through collections, delete asynchronously, compress, store the blob elsewhere and cache a pointer |
Failure mode: a cache that hides an undersized database. If the system cannot survive a cold cache, the cache is a hard dependency and needs the availability of a database: replicas, failover, and a tested warm-up plan.
Stale data and double delete
Cache-aside has a race: a reader that missed can write an old value back after a writer has deleted the key. The stale value then lives for a full TTL.
- Double delete: delete, then delete again after a few hundred ms to a second, usually from a queue so that it is retried. It narrows the window and does not close it.
- Also happens with replicas: the reader fills the cache from a lagging replica after the delete. The delay must cover replication lag, or cache fills must read the leader.
- Stronger fixes: invalidate from the change stream in commit order, or store a version with the value and only accept a set with a newer version (compare-and-set). A TTL remains the backstop.
- Order of operations: write the database, then delete the cache. Deleting first lets a reader refill the old value before the write commits.
Redis vs Memcached
| Redis | Memcached | |
|---|---|---|
| Data model | Strings, hashes, lists, sets, sorted sets, streams, bitmaps, HyperLogLog, geo | Opaque string values only |
| Threading | Commands run on one thread (I/O can be threaded), so each command is atomic | Multi-threaded: uses all cores of one node |
| Persistence | Optional snapshots and append-only file | None |
| Replication and failover | Built in: replicas, Sentinel, Cluster | None built in |
| Sharding | Redis Cluster: 16,384 hash slots assigned to nodes | Client-side, usually consistent hashing |
| Extras | Atomic scripts, transactions, pub-sub, per-key TTL, many eviction policies | LRU eviction, per-key TTL |
| Value size | Up to 512 MB per string (keep values small anyway) | 1 MB by default |
| Pick it for | Almost everything: cache plus counters, leaderboards, rate limits, sessions, locks, queues | A plain, very large, multi-threaded key-value cache with simple operations |
- Numbers: either does on the order of 100,000 simple operations per second per node, with sub-millisecond latency inside a zone. Network round trips dominate, so pipeline or batch (
MGET). - Failure mode: Redis replication is asynchronous, so a failover can lose the last acknowledged writes. Do not treat it as the system of record unless you accept that.
Sizing a cache
Access is skewed: roughly 20% of the items take 80% of the reads. Cache the hot 20% and you take most of the load off the database for a fifth of the memory.
- Check throughput too: 20,000 QPS over 3 shards is about 7,000 operations per second per node, far below what a node can do. Memory, not CPU, sets the node count here.
- Measure, then adjust: the real signal is the hit ratio and the eviction rate. Evictions with a low hit ratio mean the cache is too small or the TTL too long for the key space.
- TTL from the staleness budget, not from memory: memory pressure is handled by eviction.
Storage choices
Part 4 of 7 · pick the store from the data model and the queries
Choose a database by the shape of the data, the access patterns and the consistency needed, in that order. Most systems use more than one store: a system of record plus derived stores for search, analytics and caching. Storage engines, indexes, transactions and isolation are covered in Data Systems.
- Database families: the decision table
- SQL vs NoSQL, stated carefully
- Object storage and large files
Database families
| Family | Data model | Strengths | Weaknesses | Typical products | Pick it when the question is |
|---|---|---|---|---|---|
| Relational | Tables, rows, foreign keys, SQL | ACID transactions, joins, ad hoc queries, constraints | Write scaling beyond one node needs sharding or a distributed SQL engine | PostgreSQL, MySQL, Spanner, CockroachDB | "Do several records have to change together, correctly?" |
| Key-value | Key to opaque value | Simplest, fastest, scales by partitioning the key | Lookup by key only. No queries on the value | Redis, DynamoDB, Riak | "Is every access a get or put by one known key?" |
| Document | JSON-like documents, nested fields | Flexible schema, whole aggregate in one read, secondary indexes | Joins and multi-document invariants are weak or costly. Duplication | MongoDB, Couchbase, Firestore | "Is the data a self-contained aggregate read as a whole?" |
| Wide-column | Partition key + sorted clustering columns | Very high write throughput, linear scale, time-ordered rows in a partition | Queries must follow the key design. No joins. Usually eventual consistency | Cassandra, HBase, Bigtable, ScyllaDB | "Is it huge, write-heavy, and always queried by a known key and range?" |
| Graph | Nodes, edges, properties | Multi-hop traversals, path queries | Hard to shard. Poor at bulk aggregates | Neo4j, Neptune, JanusGraph | "Is the query about relationships several hops deep?" |
| Time-series | Timestamped points per series, with tags | Fast append, compression, downsampling, retention | High-cardinality tags are expensive. Updates are awkward | InfluxDB, TimescaleDB, Prometheus | "Is it append-only measurements queried by time window?" |
| Search | Documents in an inverted index | Full-text, relevance ranking, facets, fuzzy match | Near-real-time, not transactional. Not a system of record | Elasticsearch, OpenSearch, Solr | "Does the user type words and expect ranked results?" |
| Object / blob store | Bucket + key to immutable bytes | Cheap, very durable, effectively unlimited, HTTP access | No query, no partial update, tens of ms per request | S3, GCS, Azure Blob | "Is it a file or a large payload?" |
| Data warehouse | Columnar tables, SQL | Scans and aggregates over billions of rows | Seconds per query. Poor at point lookups and frequent small writes | BigQuery, Snowflake, Redshift, ClickHouse | "Is it analytics over history, not serving a user request?" |
Interview tip: justify each store in one sentence from the access pattern: "orders in PostgreSQL because checkout updates order, payment and stock rows in one transaction, product search in Elasticsearch fed by change events, images in object storage behind the CDN".
SQL vs NoSQL, stated carefully
The real difference is the data model and the shape in which each scales, not that one scales and the other does not. Relational databases run at very large scale with replicas and sharding. NoSQL stores scale out more easily because they give up joins and cross-partition transactions, which are exactly the features that are hard to distribute.
| Dimension | Relational | Typical NoSQL (key-value, document, wide-column) |
|---|---|---|
| Design starts from | The entities: normalise, then query any way | The queries: one table or collection per access pattern, denormalised |
| Schema | Enforced on write | Flexible, enforced by the application on read |
| Transactions | Multi-row ACID is the default | Single key or single partition by default. Wider ones are limited or cost more |
| Joins | In the database | In the application, or avoided by duplication |
| Scaling shape | Scale up, add read replicas, then shard (manual, or built in with distributed SQL) | Partitioned from day one: add nodes for capacity |
| Consistency | Strong on the leader | Often tunable or eventual |
| New query later | Add an index | May need a new table and a backfill |
- Default to relational when access patterns are still changing or invariants span records. A single well-indexed node handles thousands of writes and tens of thousands of simple reads per second, and terabytes of data.
- Choose NoSQL when the access pattern is known and simple, the write volume or data size is beyond one node, and you can live without cross-partition joins and transactions.
- The lines blur: PostgreSQL stores JSON, DynamoDB and MongoDB have transactions, and distributed SQL gives relational semantics on a partitioned store at a latency cost.
Object storage and large files
Never put large binaries in the database or send them through application servers. Store bytes in an object store, keep metadata in a database, and let clients transfer directly with short-lived signed URLs.
- Metadata vs blob split: the database row holds owner, name, size, checksum, status and the object key. It is small, queryable and transactional. The blob is immutable and addressed by key.
- Chunking: split a large file into parts (commonly 4 to 64 MB). Parts upload in parallel, a failed part is retried alone, an interrupted upload resumes, and identical chunks can be deduplicated by content hash. The file is then a list of chunk ids in the metadata store.
- Pre-signed URL: a URL signed by the service that grants one operation on one key for a few minutes. Downloads use the same idea, usually through the CDN.
- Numbers: S3 is designed for eleven nines of durability. Multipart parts are 5 MB to 5 GB with up to 10,000 parts. First-byte latency is tens of ms, so it is not a low-latency key-value store.
- Lifecycle: move cold objects to cheaper tiers by age, and delete the parts of abandoned uploads.
Failure mode: orphans. The blob is uploaded and the metadata write fails, or the reverse. Write the metadata row first with a pending status, confirm it on completion, and run a sweeper that removes pending rows and unreferenced blobs.
Scaling data
Part 5 of 7 · more copies, then more pieces
Replication copies the same data to several nodes, for availability and read capacity. Partitioning splits the data over nodes, for write capacity and size. Real systems do both: each shard is a replicated group. Consensus, quorums and consistency models are in Distributed Systems Patterns.
- Replication: leader-follower, sync vs async
- Replication lag and read-your-writes
- Failover and split brain
- Partitioning and sharding: range, hash, directory, geo
- Shard keys, hot partitions, rebalancing
- Cross-shard queries, joins and transactions
- Consistent hashing
- Secondary indexes on sharded data
Replication
In leader-follower replication one node accepts all writes and ships its log to followers, which apply it in the same order and serve reads.
| Mode | Commit is acknowledged when | Durability on leader loss | Write latency | If a follower is down |
|---|---|---|---|---|
| Asynchronous | The leader has written it | Recent writes can be lost | Lowest | No effect |
| Synchronous (all) | Every follower confirms | No loss | Slowest follower sets it | Writes stall |
| Semi-synchronous | At least one follower confirms | No loss while one confirmed copy survives | One extra round trip | Another follower takes the synchronous role, or it degrades to async |
- Use it for: surviving a node loss, serving reads near users, read scaling when reads far outnumber writes. It does not scale writes: every node applies every write.
- Numbers: same-region lag is usually milliseconds, and grows to seconds or minutes under heavy writes or long transactions on the follower. Cross-region adds the network round trip (50 to 150 ms).
- Other topologies: multi-leader (writes in several regions, conflicts must be resolved) and leaderless (quorum reads and writes) are in Distributed Systems Patterns.
Replication lag and read-your-writes
With asynchronous followers a read can arrive before the write does. The user saves a profile, reloads, and sees the old one.
| Anomaly | Fix | Cost |
|---|---|---|
| Not seeing your own write | Read the user's own data from the leader, or from the leader for a few seconds after a write | More leader load |
| Same, across devices | Position token as above, stored per user | A lookup and occasional fallback |
| Time going backwards between two reads | Monotonic reads: pin each user to one follower | Less even load. Re-pin on failure |
| Stale reads in general | Stop routing to followers whose lag exceeds a threshold | Less read capacity when lag is high |
Senior signal: say which reads may be stale. "The timeline can lag by seconds and reads from followers. Balance and settings read from the leader."
Failover and split brain
- Detection is a guess: a slow node and a dead node look the same. A short timeout causes needless failovers, a long one extends the outage. Typical: 10 to 30 s of missed heartbeats, total failover 30 s to a couple of minutes.
- Data loss: with async replication the new leader may lack the last writes. When the old leader returns, its extra writes are discarded or need manual repair.
- Split brain: a network partition leaves two nodes that each believe they are the leader, and both accept writes that then diverge.
- Prevention: promote only with a majority vote (an odd number of voters, a witness in a third zone), fence the old leader (revoke its lease, cut its storage or network), and attach an increasing epoch number to writes so that storage rejects a stale leader.
Failure mode: automatic failover during an overload. The leader is slow, not dead. Failing over moves the same load onto a colder node and adds a recovery storm. Many teams automate detection and keep a human in the loop for cross-region promotion.
Partitioning and sharding
Sharding splits one logical dataset across nodes so that each holds a part. It is the answer to write throughput or data size beyond one node, and it should come after vertical scaling, read replicas and caching.
| Scheme | Key to shard | Gain | Pay | Used for |
|---|---|---|---|---|
| Range | Contiguous key ranges (A to F, by date) | Efficient range scans. Easy to split a range | Hot spots on sequential keys: all new writes hit the last range | HBase, Bigtable, time-ordered data |
| Hash | hash(key) to a partition | Even spread | No range scans across keys. Resizing moves data unless done with fixed slots or a ring | DynamoDB, Cassandra, Redis Cluster |
| Directory / lookup | A table maps key or tenant to shard | Full control: move one tenant, isolate a big one | The lookup service is a critical dependency and must be cached and replicated | Multi-tenant SaaS, Vitess-style setups |
| Geo | User's region or country | Low latency, data residency | Uneven shard sizes. Cross-region users and queries | Regulated or globally spread user data |
Interview tip: give the arithmetic that forces sharding ("2 TB a year and 30,000 writes per second is past one node") before you shard. Sharding without a stated reason is a red flag.
Shard keys, hot partitions, rebalancing
The shard key is the hardest decision to change later. A good key has high cardinality, spreads load evenly, and is present in almost every query so that the query reaches one shard.
| Candidate key | Result |
|---|---|
user_id for user-owned data | Good: one user's data on one shard, queries are single-shard |
created_at or an auto-increment id | Bad with range partitioning: every write lands on the newest shard |
country or status | Bad: few values, heavily skewed |
tenant_id | Good until one tenant outgrows a shard. Use a directory, or a compound key |
(user_id, created_at) compound | Good: hashed first part spreads load, sorted second part gives per-user range scans |
- Hot partitions (a celebrity, a flash-sale item): add a suffix to split the key over N sub-partitions and merge on read, give the heavy key its own shard through the directory, or absorb reads with a cache.
- Rebalancing, never
hash mod N: changing N moves almost every key. Use a fixed number of logical partitions much larger than the node count (for example 1,024 or 16,384) and move whole partitions between nodes, or split ranges dynamically when they grow, or use consistent hashing. - Moving data live: copy the partition in the background, replicate changes until caught up, switch the routing entry, then delete the old copy. Throttle it: rebalancing competes with production traffic.
Cross-shard queries, joins and transactions
| Operation | Why it gets hard | What to do instead |
|---|---|---|
| Query not on the shard key | Must ask every shard. Tail latency grows with the shard count | A global index or a second copy keyed differently, or a search or analytics store |
| Join | The rows live on different nodes, so the join moves data over the network | Co-locate related tables on the same key, denormalise, or join in the application |
| Transaction | No single node can commit it. Needs two-phase commit, which blocks if the coordinator fails, or a saga | Choose the key so that a transaction stays in one shard. Otherwise a saga with compensations |
| Unique constraint, auto-increment | No shard sees all the rows | Include the shard key in the constraint. Generate ids outside the database (see unique IDs) |
| Sorting and paging | Each shard returns its top N, and deep pages multiply the work | Cursor paging on the shard key, or a precomputed view |
Trade-off: sharding buys write scale and pays with every feature that needs a global view. Two-phase commit and sagas are covered in Distributed Systems Patterns.
Consistent hashing
Consistent hashing maps keys to nodes so that adding or removing a node moves only about 1 / N of the keys, where hash mod N moves nearly all of them.
How it works: hash both nodes and keys onto the same circular space (0 to 232 - 1). A key belongs to the first node found going clockwise from the key's position.
| Event | What moves | What does not |
|---|---|---|
| Node D joins at 55 | Keys in (40, 55] move from C to D. The key at 60 stays on C | Everything on A and B |
| Node B leaves | Keys in (10, 40] move from B to C, the next node clockwise | Everything on A, and C's existing keys |
Same change with hash mod N | About N / (N + 1) of all keys change node | Almost nothing |
- Problem with one point per node: arcs are uneven, so load is uneven, and a departing node dumps its whole arc onto one neighbour, which can then fail in turn.
- Virtual nodes: place each physical node at many positions (100 to 200 is common). Arcs average out, a leaving node's keys spread over all the others, and a bigger machine can be given more positions.
- Replication on the ring: store each key on the next R distinct physical nodes clockwise. This is the preference list in Dynamo-style stores.
- Where it is used: Dynamo and Cassandra partitioning, client-side sharding of Memcached (ketama), cache-affinity load balancing, CDN request routing.
- Alternatives: rendezvous hashing (score every node per key and take the highest: simple, no ring), jump consistent hash (no memory, nodes can only be added or removed at the end), and bounded-load variants that cap any node at a factor above the average.
Senior signal: consistent hashing limits how much data moves. It does not fix a hot key, which lands on one node whatever the ring looks like.
Secondary indexes on sharded data
Data is sharded by one key and queried by another (orders sharded by user_id, looked up by restaurant_id). The index can be kept per shard or as a separate partitioned structure.
| Local (partitioned by document) | Global (partitioned by term) | |
|---|---|---|
| Write | One shard, atomic with the row | Row shard + index partition: distributed, usually asynchronous |
| Read by indexed field | Scatter-gather over all shards | One index partition, then the rows |
| Consistency | Index always matches the row | Index can lag the row |
| Use when | The query also carries the shard key, or is rare | The query is frequent and does not carry the shard key |
| Examples | Elasticsearch shards, MongoDB, Cassandra secondary index, DynamoDB local secondary index | DynamoDB global secondary index |
Asynchronous work
Part 6 of 7 · take work off the request path
A queue or a log between two components decouples them in time: the producer returns at once, the consumer works at its own pace, and a burst becomes a backlog instead of an outage. The price is eventual completion and duplicate handling.
Message queues vs event streams
A queue hands each message to one worker and forgets it once acknowledged. A stream is an append-only log that keeps every event for a retention period, and each reader keeps its own position.
| Message queue | Event stream / log | |
|---|---|---|
| Unit | A task to do once | A fact that happened |
| Ordering | Best effort, or FIFO within a group at lower throughput | Total order within a partition |
| Retention | Until acknowledged | By time or size (days, or forever), whether or not it was read |
| Consumer model | Competing consumers, per-message acknowledgement, redelivery on timeout | Consumer groups that commit an offset per partition |
| Replay | No | Yes: rewind the offset. New consumers can read history |
| Per-message features | Delay, priority, routing rules, per-message retry and dead-lettering | Few: one slow message blocks its partition |
| Throughput | Thousands to tens of thousands per second | Hundreds of thousands to millions per second per cluster |
| Products | RabbitMQ, Amazon SQS, ActiveMQ | Apache Kafka, Amazon Kinesis, Pulsar |
| Use for | Background jobs, emails, work distribution | Event sourcing, change data capture, analytics pipelines, many consumers of the same events |
- Why a log is fast: sequential appends to disk, batching, and consumers that read straight from the page cache. This is Kafka's documented design.
- Payload size: keep messages small (Kafka's default limit is about 1 MB). Put large payloads in object storage and send the key.
Delivery semantics
| Guarantee | How it is achieved | Failure outcome | Use for |
|---|---|---|---|
| At-most-once | Acknowledge (or commit the offset) before processing. No retries | Message can be lost, never duplicated | Metrics, logs, anything where a gap is harmless |
| At-least-once | Process, then acknowledge. Redeliver on timeout | Never lost, can be delivered twice | The default. Requires idempotent consumers |
| Effectively-once | At-least-once delivery plus deduplication or idempotent writes, or a transaction that covers the result and the offset | Duplicates arrive and have no effect | Payments, counters, inventory |
Senior signal: "exactly-once delivery" does not exist over an unreliable network. What exists is exactly-once effect: at-least-once delivery plus an idempotency key that the consumer checks. Idempotency keys, the outbox pattern and transactional messaging are in Distributed Systems Patterns.
Consumer groups and partitions
A topic is split into partitions. The producer picks the partition from the hash of a key. Within a consumer group each partition is read by exactly one consumer, which is what preserves order per key.
- Parallelism is capped by the partition count: a group with more consumers than partitions has idle consumers. Choose the count for peak throughput with headroom. It can be increased later, and doing so changes which partition a key maps to.
- Rebalance: when a consumer joins or dies, partitions are reassigned and consumption pauses briefly. Messages processed and not yet committed are delivered again to the new owner.
- Lag (latest offset minus committed offset) is the health metric. Alert on it and autoscale on it.
- Durability: each partition is replicated (typically 3 copies). A producer that waits for all in-sync replicas trades latency for no loss on a broker failure.
Dead-letter queues, backpressure, ordering keys
- Dead-letter queue: holds poison messages with their error and attempt count. Alert on its depth, fix the cause, then redrive. Retry with exponential backoff and jitter first. On a stream, publish the failed event to a retry or dead-letter topic and move on, which gives up strict ordering for that key.
- Backpressure: what happens when producers outrun consumers. Options, in order of preference: scale consumers on lag, bound the queue and reject or slow producers (HTTP 429, blocking send), shed low-priority work, drop the oldest. An unbounded queue only converts overload into unbounded latency and then an out-of-memory failure.
- Ordering key: messages with the same key (order id, account id) go to the same partition or message group and are processed in order. Different keys run in parallel. Choose the narrowest key that needs ordering: ordering everything globally means one partition and no parallelism.
Failure mode: retries on an overloaded downstream multiply the load. Cap attempts, back off, and stop consuming (circuit breaker) when the dependency is down, letting the backlog wait in the queue.
Pub-sub and fan-out
In publish-subscribe the producer publishes to a topic without knowing who listens, and every subscriber gets its own copy.
| Fan-out choice | How | Gain | Pay |
|---|---|---|---|
| On write (push) | At publish time, copy the item to every recipient's inbox or queue | Reads are a single cheap lookup | Write cost grows with the number of recipients. A sender with millions of followers is a storm |
| On read (pull) | Store once. Each reader assembles its view at read time | Cheap writes | Slow, expensive reads |
| Hybrid | Push for ordinary senders, pull for the few with huge audiences | Bounded write cost and fast reads | Two code paths to merge |
- Products: SNS + SQS, Google Pub/Sub, Kafka topics with one consumer group per subscriber, Redis pub-sub (not durable: a subscriber that is offline misses the message).
Task queues and schedulers
A task queue runs background jobs on a worker pool. A scheduler decides when a job becomes due and hands it to the queue.
- Task contract: idempotent (it will run twice some day), with a timeout, bounded retries with backoff, and a visibility timeout longer than the task's worst case so that a second worker does not pick it up mid-run.
- Scheduler designs: poll a database table for
run_at <= nowwith row locking (simple, good to thousands of jobs per second), delayed messages in the queue, or a sorted set scored by due time. For long delays keep jobs in durable storage and load only the next few minutes into memory. - Exactly one scheduler firing: run several scheduler instances for availability and let one lead (leader election), or let all poll and claim jobs with an atomic update.
- Isolation: separate queues per priority or job class so that a flood of bulk jobs cannot delay password-reset emails.
- Products: Celery, Sidekiq, SQS with workers, Temporal or Airflow for multi-step workflows.
Specialised blocks
Part 7 of 7 · named components for recurring sub-problems
These come up inside almost every case study. Knowing the standard options lets you name one in a sentence and spend the time on the trade-off.
- Rate limiter: five algorithms and where to place it
- Unique ID generation
- Bloom filter and other probabilistic structures
- Search index
- Geospatial indexing
- Blob and media processing pipeline
- Heartbeats, health checks, locks and leaders
Rate limiter
A rate limiter caps how many requests a client (user, API key, IP address) may make in a period, to protect capacity and enforce fairness and quotas.
| Algorithm | How | Bursts | Memory per key | Accuracy and notes |
|---|---|---|---|---|
| Token bucket | Tokens refill at rate r up to capacity b. A request spends one | Allowed, up to b | 2 values: tokens, last refill time | The usual default. Two knobs: sustained rate and burst size |
| Leaky bucket | Requests enter a queue that drains at a fixed rate. Overflow is dropped | Smoothed away | Queue or 2 values | Constant output rate. Adds delay. Good in front of a fragile backend |
| Fixed window | One counter per window (per minute) | Up to 2x the limit across a window boundary | 1 counter | Simplest. Fine for coarse quotas |
| Sliding window log | Store the timestamp of every request, count those inside the window | None beyond the limit | One entry per request, up to the limit | Exact and expensive |
| Sliding window counter | Current window count + previous window count weighted by overlap | Small error | 2 counters | Approximate, cheap. Good default for strict limits at scale |
- Distributed counting: keep counters in Redis and make check-and-update atomic (a script or a single increment with expiry). For very high rates, each node takes a local batch of tokens and syncs periodically, trading exactness for no round trip.
- Layers: edge or gateway for per-client limits, the service for per-tenant business quotas, the client library for polite backoff.
- When the limiter's store is down: fail open for availability (most APIs), fail closed for abuse-sensitive paths such as login or one-time-password sending.
Interview tip: state the key, the limit and the response: "100 requests per minute per API key, token bucket with burst 20, 429 with Retry-After, limits returned in response headers".
Unique ID generation
| Scheme | Size | Sortable by time | Coordination | Gain | Pay |
|---|---|---|---|---|---|
| Database auto-increment | 64 bits | Yes | The one database | Simple, compact, dense | Single writer. Breaks with sharding. Exposes volume and is guessable |
| UUID v4 | 128 bits | No | None | Generate anywhere, no collisions in practice (122 random bits) | Large. Random inserts scatter across a B-tree index and hurt write locality |
| UUID v7 | 128 bits | Yes (ms) | None | Timestamp prefix keeps inserts at the end of the index | Large. Reveals creation time |
| Snowflake-style | 64 bits | Yes (ms) | A machine id per generator | Compact, ordered, generated locally at very high rates | Depends on clocks. Machine ids must be assigned |
| Ticket server | 64 bits | Roughly | A dedicated id service | Simple numeric ids across shards | A network call per id or per block, and a service to keep highly available |
The Snowflake layout, as published by Twitter. Variants move the bit boundaries.
| Field | Bits | Holds | Capacity |
|---|---|---|---|
| Sign | 1 | Always 0, so the id is positive in signed types | |
| Timestamp | 41 | Milliseconds since a custom epoch | About 69 years |
| Machine id | 10 | Datacenter + worker | 1,024 generators |
| Sequence | 12 | Counter within the same millisecond | 4,096 ids per ms per generator |
- Clock moving backwards could repeat an id. The generator must refuse or wait until the clock passes the last timestamp it used.
- Ticket server: a database that only hands out auto-increment values. Run two with different offsets (one odd, one even) for availability, and hand out blocks of ids (1,000 at a time) to cut round trips. The ids are then only roughly ordered.
- Choosing: one database: auto-increment. Sharded or generated by many services, with ordering needed: Snowflake-style or UUID v7. Ids created offline or on clients: UUID. Public ids should not be guessable, so do not expose a raw sequence.
Bloom filter and friends
A Bloom filter is a bit array with k hash functions that answers "is this key in the set?" with no false negatives: "no" is certain, "yes" means probably.
- Sizing intuition: about 10 bits per key with 7 hash functions gives a 1% false-positive rate, whatever the key size. Each further 10x reduction costs about 4.8 more bits per key. One billion keys at 1% is about 1.2 GB.
- Limits: no deletes (a counting variant allows them at several times the memory), no listing, and the rate rises once more keys are added than it was sized for.
- Uses: skipping disk reads for absent keys in LSM-tree stores, blocking cache penetration, "already seen" checks for crawled URLs or recommended items, username-taken pre-checks.
- HyperLogLog estimates the number of distinct items (unique visitors) to within about 1% in roughly 12 KB. Count-Min Sketch estimates how often each item occurred (top-K, heavy hitters) in fixed memory, and can only overestimate.
Search index
An inverted index maps each term to the sorted list of documents that contain it (a posting list), so a text query becomes a few list lookups and an intersection instead of a scan.
- Ranking: a relevance score such as BM25 (term frequency, rarity of the term, document length), usually blended with business signals such as popularity and recency.
- Writes: new documents go to small immutable segments that are merged in the background, so search is near-real-time (about a second behind), not transactional.
- Scaling: the index is split into shards. A query is scatter-gather over all shards, each returning its top N.
- Use when users search text, need facets or fuzzy matching. Avoid as the system of record: feed it from the primary database through change events and be able to rebuild it.
Geospatial indexing
"Find things near this point" is a two-dimensional query, and ordinary indexes are one-dimensional. Every scheme maps an area to a cell id that an ordinary index can store.
| Geohash | Quadtree | S2 / H3 | |
|---|---|---|---|
| Idea | Interleave latitude and longitude bits into a base-32 string. A shared prefix means the same cell | Tree that splits a square into 4 whenever it holds too many points | S2: square cells on a sphere ordered along a space-filling curve, 64-bit ids. H3: hexagonal cells |
| Cell size | Fixed per length: 5 chars about 5 km, 6 about 1 km, 7 about 150 m | Adapts to density: small cells downtown, large in the countryside | Fixed per level, many levels |
| Storage | Any database or Redis: a string or integer column with a normal index | Usually an in-memory structure you build and update | An integer column, with a library to compute cells and coverings |
| Gain | Simplest. Prefix queries. Built into Redis and Elasticsearch | Even load per leaf. Natural "k nearest" search | Little distortion. S2 covers arbitrary regions well. H3 neighbours are all at the same distance |
| Pay | Cell edges, and cells shrink toward the poles. Dense areas overload a cell | Rebuilding and rebalancing as points move. Harder to shard | Library dependency, more concepts |
| Pick for | Nearby search with a fixed radius | Very uneven density, static or slowly changing points | Ride-hailing and maps at scale, region coverage, aggregation by area |
- Moving objects (drivers): keep the latest position in memory keyed by cell with a short TTL, updated every few seconds. Do not write each update to the durable store.
Blob and media processing pipeline
Uploads are processed asynchronously: the original lands in object storage, an event starts a set of workers, and the derived files are published through the CDN.
- Work as a graph: split a video into segments, transcode each segment into several resolutions and bitrates in parallel, then write the manifests. Images get resized variants and a stripped copy without location metadata.
- Each step is idempotent and writes to a deterministic output key, so a retry overwrites instead of duplicating.
- Status: the client polls the metadata service or receives a webhook or push when the state becomes ready.
- Scaling: autoscale workers on queue lag. Use separate queues for short and long jobs so that a 4-hour video does not delay profile pictures.
- Numbers: transcoding is CPU or GPU bound and takes a meaningful fraction of the video's duration per rendition. It is the cost centre of the pipeline.
Heartbeats, health checks, locks and leaders
A heartbeat is a periodic "I am alive" signal. A node that misses several in a row is treated as dead, which is always a guess: the node may only be slow or cut off.
| Check | Asks | On failure | Should test |
|---|---|---|---|
| Liveness | Is the process stuck? | Restart it | Only the process itself. Never dependencies |
| Readiness | Can it serve traffic now? | Take it out of rotation, do not restart | Warm-up done, critical local resources available |
| Deep / dependency | Are the things it calls healthy? | Alert, degrade features | Used for dashboards, not for ejecting instances |
- Push vs pull: nodes send heartbeats to a registry (works across network boundaries, the registry must expire entries), or a checker probes the nodes (no client code, the checker must reach everyone).
- At scale: gossip protocols spread liveness information between peers without a central monitor.
- Distributed lock: a lease with an expiry and a fencing token, so that only one holder acts at a time. See Distributed Systems Patterns.
- Leader election: one node from a group is chosen through a consensus-backed store (etcd, ZooKeeper) and keeps the role by renewing a lease. See Distributed Systems Patterns.
Cheat sheet
| Need | Block | Key trade-off |
|---|---|---|
| Route users to the nearest healthy region | Geo or latency DNS, anycast | Failover is only as fast as the TTL |
| Serve static or shared content fast | CDN | Invalidation delay. Use versioned URLs |
| Spread requests, hide dead servers | L7 load balancer | Extra hop, and it must itself be redundant |
| Auth, quotas, routing in one place | API gateway | Central choke point. Keep logic out of it |
| Handle more request traffic | Stateless services + autoscaling | State moves to shared stores, which become the bottleneck |
| Too many database connections | Connection pool or pooling proxy | Pool exhaustion when queries are slow |
| Cut read latency and database load | Cache-aside on Redis | Staleness, and survival of a cold cache |
| Absorb very frequent small writes | Write-behind | Loses unflushed writes on a crash |
| Hot key expiring under load | Lock, coalesce, jitter, early refresh | Waiters see extra latency or stale data |
| Lookups for keys that do not exist | Negative cache, Bloom filter | False positives, and no deletes |
| Transactions and flexible queries | Relational database | Write scaling needs sharding |
| Huge write volume by known key | Wide-column or key-value store | Queries are fixed by the key design |
| Files, images, video | Object store + pre-signed URLs + CDN | No queries. Metadata lives elsewhere |
| Survive a node loss, scale reads | Leader-follower replication | Lag, and data loss on async failover |
| Scale writes or data size | Sharding on a well-chosen key | Cross-shard joins, transactions and queries |
| Add or remove nodes with little data movement | Consistent hashing with virtual nodes | Does not help with hot keys |
| Query sharded data by another field | Global secondary index | Index lags the row. Extra write cost |
| Run slow work later | Message queue + workers | At-least-once: handlers must be idempotent |
| Many consumers, replay, ordered events | Event log with consumer groups | Order only within a partition. Parallelism capped by partitions |
| Notify several systems of one event | Pub-sub, one queue per subscriber | Fan-out on write grows with the audience |
| Messages that always fail | Retries with backoff, dead-letter queue | Someone has to watch and redrive it |
| Protect an API from abuse or overload | Token bucket at the gateway | Shared counter store. Fail open or closed |
| IDs across shards, time-ordered | Snowflake-style or UUID v7 | Clock dependence, or 128-bit keys |
| Full-text search | Inverted index | Near-real-time, a derived store to keep in sync |
| Find things nearby | Geohash, quadtree, S2 or H3 | Cell-edge handling, uneven density |
| Process uploads | Object store event, queue, workers | Asynchronous: the client must track status |
| Know which nodes are alive | Heartbeats and health checks | Detection speed versus false positives |
Numbers to keep in your head, all approximate:
| Thing | Order of magnitude |
|---|---|
| Redis or Memcached operation, same zone | 0.5 to 1 ms, about 100,000 operations per second per node |
| CDN edge hit vs cross-continent origin | 10 to 30 ms vs 100 to 300 ms |
| Object store first byte | Tens of ms |
| Replication lag, same region / cross region | Milliseconds normally / 50 to 150 ms and up |
| Database failover | 30 s to a couple of minutes |
| Consistent hashing, node added to N | About 1 / (N + 1) of keys move |
| Bloom filter | 10 bits per key for 1% false positives |
| Snowflake id | 41 + 10 + 12 bits: 69 years, 1,024 generators, 4,096 ids per ms each |
| Kafka message limit by default | About 1 MB |
Next: how these stores work inside, from storage engines and indexes to transactions and isolation, in Data Systems.