System design, part 1 of 12
System Design Foundations
The map of the whole path, a repeatable 45-minute interview structure, the estimation arithmetic and the core trade-offs that every later page relies on. Written for an experienced backend engineer: definitions are precise, numbers are order of magnitude, and each chapter leads with the decision.
The map
Part 1 of 5 · what is asked and where it lives on this site
System design interviews come in two shapes: high-level design (HLD) of a distributed system, and low-level design (LLD) of the classes inside one service. Everything else on this path is the toolkit those two rounds draw on.
- HLD, LLD and the supporting areas: what each round asks and scores
- How the 12 pages fit: reading order and dependencies
HLD, LLD and the supporting areas
HLD answers "which components, how does data flow, how does it scale and fail". LLD answers "which classes and interfaces, which patterns, how is it made thread-safe and extensible". A senior candidate is expected to move between both levels in the same conversation.
| Round type | What is asked | What is evaluated | Where on this site |
|---|---|---|---|
| HLD (system design) | "Design a news feed / chat / rate limiter" for N users | Requirement scoping, numbers, component choice, data model, scaling, failure handling, trade-offs | HLD Case Studies: Core, HLD Case Studies: Advanced |
| LLD / machine coding | "Design a parking lot / splitwise / LRU cache" as classes, often with running code | Object modelling, interfaces, patterns, SOLID, concurrency, testability, extensibility | Low-Level Design, LLD Case Studies |
| Building-block probes | "How does a load balancer / cache / queue behave when ..." | Knowing each component's guarantees, limits and failure modes | Building Blocks |
| Storage deep dive | Choice of database, indexing, replication, partitioning, transactions | Matching the store to access patterns; consistency reasoning | Data Systems |
| Distributed-systems deep dive | Consensus, idempotency, ordering, clocks, distributed transactions | Correctness under partial failure | Distributed Systems Patterns |
| API design | REST / gRPC / events, pagination, versioning, real-time channels | Clean contracts, backward compatibility, idempotent writes | API and Communication Design |
| Architecture discussion | Monolith vs microservices, event-driven, CQRS, migration paths | Judgement: fitting structure to team and load, not fashion | Architecture Styles |
| Operations and reliability | SLOs, observability, deploys, overload, disaster recovery | Operational maturity: you have run systems in production | Reliability and Operations |
| The loop itself | How to run the session, question bank, study plan | Driving, prioritising, communicating | Senior Interview Playbook |
Senior signal: a mid-level answer lists components. A senior answer states the requirement that forces each component, the number that sizes it and what happens when it fails.
How the 12 pages fit
Read top to bottom. The toolkit pages can be read in any order after this page; the case studies assume all of them.
- Short on time: this page, Building Blocks, Data Systems, then the case studies.
- LLD round only: Low-Level Design then LLD Case Studies.
The interview framework
Part 2 of 5 · a repeatable 45-minute structure
The framework exists so that no minute is spent deciding what to do next. Requirements drive numbers, numbers drive the API and data model, and those drive the diagram.
The eight steps and the time budget
| Minutes | Step | Budget | Output on the board |
|---|---|---|---|
| 0 to 5 | 1. Clarify requirements | 5 | 3 to 5 functional, 3 to 5 non-functional, out-of-scope list |
| 5 to 9 | 2. Estimate | 4 | QPS (read, write, peak), storage, the one number that shapes the design |
| 9 to 13 | 3. API | 4 | 4 to 6 endpoints or RPCs with key parameters |
| 13 to 17 | 4. Data model | 4 | Entities, keys, main access patterns, store choice |
| 17 to 27 | 5. High-level diagram | 10 | End-to-end path for each core use case |
| 27 to 39 | 6. Deep dives | 12 | 1 to 2 areas taken to implementation depth |
| 39 to 43 | 7. Bottlenecks, failures, trade-offs | 4 | Single points of failure, hot spots, what was traded away |
| 43 to 45 | 8. Wrap-up | 2 | Summary against requirements, what you would do next |
Interview tip: a working end-to-end design by minute 27 matters more than a perfect step 2. If the interviewer says "skip the numbers", skip them and state your assumed scale in one sentence.
Functional vs non-functional requirements
Functional: what the system does ("a user can post", "followers see the post"). Non-functional: how well it does it (scale, latency, availability). Functional requirements pick the boxes; non-functional requirements pick how many, where and with which guarantees.
| Non-functional | Question to ask | What the answer changes |
|---|---|---|
| Scale | DAU, read:write ratio, data size, growth, peak vs average? | Single node vs partitioned; cache and replica count |
| Latency | p99 target for the hot path? Is it user-facing or batch? | Sync vs async, caching, precomputation, region placement |
| Availability | Target nines? Is degraded service acceptable? | Redundancy level, multi-AZ or multi-region, fallbacks |
| Consistency | Can a read be stale? For how long? Which operations cannot be? | Leader reads vs replica reads, quorum, transactions |
| Durability | Is losing an acknowledged write ever acceptable? | Sync replication, fsync policy, backups, queue persistence |
| Security | Who may call it? Any PII, payments, multi-tenant isolation? | AuthN/AuthZ placement, encryption, audit trail |
| Cost | Is there a budget or cost-per-request constraint? | Storage tiering, compute choice, how much is precomputed |
| Compliance | Data residency, retention, right to delete? | Region pinning, deletion pipelines, immutable logs |
Interview tip: do not ask all eight. Ask scale, latency and consistency, then propose the rest: "I'll assume 99.9% availability and no acknowledged write is lost; tell me if that is wrong".
What to produce and say at each step
| Step | Produce | Say | Common mistake |
|---|---|---|---|
| 1. Clarify | Three short lists: functional, non-functional, out of scope | "The core use cases are A, B, C. I'm leaving out D and E unless you want them." | Designing before scoping; 10 minutes of questions with no decisions |
| 2. Estimate | Read and write QPS, peak, storage per year, one design-shaping number | "100:1 read-heavy, 12k reads/s at peak, so caching and replicas matter more than write scaling." | Arithmetic with no conclusion; false precision |
| 3. API | Endpoints with inputs, outputs, auth, idempotency key on writes, pagination on lists | "These map one-to-one to the functional requirements." | CRUD for every entity instead of the use cases; no pagination |
| 4. Data model | Entities, primary keys, indexes for each access pattern, SQL vs NoSQL with a reason | "The hot query is by user_id ordered by time, so that is the partition and sort key." | Picking the database before listing queries |
| 5. High-level | Client, edge, services, stores, async paths; trace each use case through it | "Let me walk a write, then a read, through the diagram." | Boxes with no data flow; adding Kafka with no stated need |
| 6. Deep dives | Detail on the 1 to 2 hardest parts: schema, algorithm, protocol, failure handling | "The risky part is the fan-out. Shall I go deep there, or do you prefer the storage layer?" | Shallow coverage of everything; waiting to be asked |
| 7. Bottlenecks | SPOFs, hot keys, thundering herds, what breaks at 10x, monitoring | "If this cache cluster is lost, the DB sees 20x load; I'd protect it with ..." | Claiming the design has no weaknesses |
| 8. Wrap-up | 30-second recap mapped to requirements; next steps | "We meet the latency goal via X; the open risk is Y; next I would add Z." | Trailing off; introducing new components at minute 44 |
API and data model sketch
The level of detail to aim for on the board, here for a short-post service. Enough to show keys, idempotency and pagination; no more.
Estimation
Part 3 of 5 · back-of-envelope arithmetic and the numbers to memorise
Estimation is not about the answer; it is about finding which resource is the constraint. Round hard, keep one significant figure, and end every estimate with a design consequence.
The method
| Quantity | Formula | Shortcut |
|---|---|---|
| Average QPS | DAU x actions per user per day / 86,400 | Divide by 10^5: 1M per day is about 12/s, call it 10 |
| Peak QPS | Average x peak factor | 2x to 3x typical; 10x or more for flash sales and live events |
| Storage per day | Writes per day x bytes per record | 1M x 1 KB = 1 GB |
| 5-year storage | Per day x 365 x 5 x replication factor | x 2,000 per replica; x 3 replicas is about x 5,500 |
| Bandwidth | QPS x response size | 1,000/s x 1 MB = 1 GB/s = 8 Gbps |
| Cache memory | 20% of the hot working set x object size (80/20 rule) | Plus about 30% to 50% overhead for keys and metadata |
| Servers | Peak QPS / QPS per server, plus headroom | Add 30% and N+1 or N+2 for failures and deploys |
80/20 rule: a rule of thumb that about 20% of the objects receive about 80% of the reads. Real workloads are often more skewed (Zipf-like), which makes caching work even better and hot keys worse.
Powers of two and time
| Power | Approximate value | Unit | Feel for it |
|---|---|---|---|
| 2^10 | 1 thousand (10^3) | 1 KB | A short JSON record or a row |
| 2^20 | 1 million (10^6) | 1 MB | A compressed photo is 0.2 to 2 MB |
| 2^30 | 1 billion (10^9) | 1 GB | 1M records of 1 KB |
| 2^40 | 1 trillion (10^12) | 1 TB | 1B records of 1 KB; one large disk |
| 2^50 | 10^15 | 1 PB | Object-storage territory |
| 2^32 | about 4.3 billion | IPv4 space; 32-bit id space | |
| 2^64 | about 1.8 x 10^19 | 64-bit id space; never runs out |
| Time fact | Value | Use |
|---|---|---|
| Seconds in a day | 86,400, round to 10^5 | Per-day to per-second |
| Seconds in a month | about 2.6 million (2.5 x 10^6) | Monthly quotas |
| Seconds in a year | about 31.5 million (pi x 10^7) | Downtime budgets |
| 1 request/s | about 86k per day, 2.6M per month | Sanity checks |
| Days in 5 years | about 1,800, round to 2,000 | Storage horizon |
| Object | Typical size (approximate) |
|---|---|
| UUID / 64-bit id / timestamp | 16 bytes / 8 bytes / 8 bytes |
| Tweet-sized text with metadata | about 1 KB |
| Web page HTML | about 100 KB |
| Photo (compressed) | about 200 KB to 2 MB |
| 1 minute of video (720p to 1080p, compressed) | about 10 to 50 MB |
Worked example
A short-post service: 100M DAU, each user writes 2 posts and loads the feed 20 times a day. A post is 1 KB; 10% carry a 500 KB image; a feed page returns 10 posts. In the interview you would round as in the right-hand column of the table below; the code shows the exact arithmetic.
| Quantity | Interview rounding | Design consequence |
|---|---|---|
| Write QPS | 200M / 10^5 = 2k/s | One leader per shard can take it; writes are not the hard part |
| Read QPS | 2B / 10^5 = 20k/s, peak 60k/s | 10:1 read-heavy: precompute feeds, cache, read replicas |
| Text storage | 200 GB/day, about 1 PB in 5 years with 3 replicas | Far beyond one node: partition by user or post id from the start |
| Media storage | 10 TB/day, about 20 PB in 5 years | Object storage plus CDN; never in the database |
| Egress (text) | 20k/s x 10 KB = 200 MB/s, about 2 Gbps | Fine; images dominate bandwidth, so serve them from the CDN |
| Cache | 20% of 200 GB = 40 GB, 60 GB with overhead | A small replicated cache cluster, not a fleet |
| App servers | 60k / 1k per server = 60, about 80 to 90 with headroom | Stateless tier behind a load balancer, autoscaled |
Senior signal: the last column. Every number ends in "so we need ..." or "so this is not a concern". The 1,000 QPS per server figure is an assumption; say so and move on.
Latency numbers
Order of magnitude only; they vary by hardware and year. What matters is the ratios: memory is about 1,000x faster than SSD, a network hop inside a datacentre costs about as much as thousands of memory reads, and crossing an ocean costs hundreds of datacentre hops.
| Operation | Time (order of magnitude) | Relative |
|---|---|---|
| L1 cache reference | about 1 ns | 1x |
| Branch mispredict / L2 cache reference | about 3 to 10 ns | 10x |
| Mutex lock and unlock (uncontended) | about 20 ns | 20x |
| Main memory (RAM) reference | about 100 ns | 100x |
| Compress 1 KB with a fast codec | about 2 to 10 us | 10^4 x |
| SSD random read | about 100 us | 10^5 x |
| Read 1 MB sequentially from memory | about 10 to 100 us | 10^5 x |
| Round trip within the same datacentre | about 0.5 ms | 5 x 10^5 x |
| Read 1 MB sequentially from SSD | about 1 ms | 10^6 x |
| Spinning-disk seek | about 10 ms | 10^7 x |
| Read 1 MB sequentially from spinning disk | about 10 to 20 ms | 10^7 x |
| Round trip between continents | about 100 to 150 ms | 10^8 x |
- A cache hit over the network (about 0.5 to 1 ms) is dominated by the network, not by the cache.
- Sequential I/O beats random I/O by orders of magnitude on disk, which is why logs (WAL, Kafka, LSM trees) append.
- A cross-region synchronous call adds about 100 ms per round trip; keep them off the hot path or pay for it knowingly.
- Human-perceived "instant" is about 100 ms; a typical API p99 budget is 100 to 500 ms.
Throughput of one node
Approximate, for sizing only. Real figures depend heavily on hardware, payload size, query shape and durability settings; state the assumption and offer to load-test.
| System (one node) | Approximate throughput | Limited by |
|---|---|---|
| Application server, typical business logic | about 1k requests/s (hundreds to a few thousand) | CPU per request, downstream latency, runtime |
| Reverse proxy / L7 load balancer | tens of thousands of requests/s | TLS handshakes, connections, CPU |
| Relational DB, indexed point reads | about 10k queries/s | Buffer-pool hit rate, connections |
| Relational DB, transactional writes | about 1k to 5k writes/s | WAL fsync, index maintenance, locks |
| In-memory key-value store (Redis-like) | about 100k operations/s | Single-threaded command loop, network |
| Log broker (Kafka-like) | hundreds of thousands of small messages/s, on the order of 100 MB/s | Disk and network throughput, replication |
| LSM wide-column node (Cassandra-like) | about 10k writes/s | Compaction, disk |
| Search node (Lucene-based) | hundreds to low thousands of queries/s | Query complexity, shard count |
- Connections: one server can hold on the order of 100k to 1M mostly idle connections (WebSocket, long poll) if memory per connection is small; active throughput is the tighter limit.
- Disk: a comfortable single-node dataset is up to a few TB; past that, rebuild and backup times force partitioning before capacity does.
Availability nines
| Availability | Downtime per year | Per 30-day month | Typically needs |
|---|---|---|---|
| 99% (two nines) | 3.65 days | 7.2 hours | One server and a restart script |
| 99.9% (three nines) | 8.76 hours | 43 minutes | Redundant instances, automated failover |
| 99.95% | 4.38 hours | 22 minutes | Multi-AZ, tested failover |
| 99.99% (four nines) | 52.6 minutes | 4.3 minutes | No manual step in recovery; safe deploys; multi-AZ at least |
| 99.999% (five nines) | 5.26 minutes | 26 seconds | Multi-region active-active; very few systems truly need it |
Interview tip: at four nines a human cannot be in the recovery loop: paging and logging in already takes longer than the monthly budget.
Core trade-offs
Part 4 of 5 · the vocabulary of every design decision
Each chapter is one axis you will be asked to choose a position on. The detail (replication protocols, quorums, consensus) lives in Data Systems and Distributed Systems Patterns.
- Scalability: vertical vs horizontal
- Latency, throughput and percentiles
- Availability arithmetic
- Reliability vs availability vs durability
- Consistency models as a ladder
- CAP and PACELC
- ACID vs BASE, strong vs eventual
- Read-heavy vs write-heavy
- Push vs pull, sync vs async, stateful vs stateless
- Single-leader vs multi-leader vs leaderless
Scalability: vertical vs horizontal
Scale up (bigger machine) until it stops being cheap or safe; scale out (more machines) when you need capacity beyond one box or redundancy. Scaling out is easy for stateless tiers and hard for state.
| Vertical (scale up) | Horizontal (scale out) | |
|---|---|---|
| Gain | No code change, no distributed-systems problems, strong consistency stays easy | Near-linear capacity, redundancy, incremental cost |
| Pay | Hard ceiling, single point of failure, downtime to resize, price grows faster than capacity at the top end | Load balancing, partitioning, coordination, partial failure, operational complexity |
| Use when | Databases early on; anything under one node's limits | Stateless services always; data tier once one node is not enough |
- Stateless service: any instance can serve any request because session and data live in a shared store or in the request (token). This is what makes load balancing, autoscaling and rolling deploys trivial.
- Scalability is not performance: a system is scalable if adding resources adds capacity roughly in proportion.
Senior signal: "one large Postgres with a replica handles this for the first two years; I'd shard only when write volume or dataset size forces it" is a stronger answer than sharding on day one.
Latency, throughput and percentiles
Latency is time per request; throughput is requests per unit time. Batching and queueing raise throughput and raise latency. Near saturation, latency grows non-linearly, so plan to run at 50% to 70% utilisation.
| Metric | Meaning | Use |
|---|---|---|
| p50 (median) | Half of requests are faster | Typical experience; hides problems |
| p95 | 1 in 20 is slower | Capacity planning |
| p99 | 1 in 100 is slower | SLOs; heavy users hit it constantly |
| p99.9 | 1 in 1,000 is slower | Large fan-out systems |
| Average | Sum / count | Avoid: one 10 s outlier hides in a 50 ms mean |
- If each backend exceeds its p99 on 1% of calls, a request touching 100 backends is slow with probability 1 - 0.99^100 = 63%. With 10 backends it is about 10%.
- Tail-latency tools: timeouts with budgets, hedged requests (send a second copy after the p95 delay), partial results, fewer hops, avoiding shared queues.
- Sources of tail latency: GC pauses, lock contention, cold caches, noisy neighbours, retries, queue build-up.
Interview tip: always quote a percentile: "p99 under 200 ms", never "fast" or "200 ms average".
Availability arithmetic
Components in series (all must work) multiply availabilities and make things worse. Components in parallel (any one is enough) multiply failure probabilities and make things better, assuming failures are independent.
| Step | Arithmetic | Result |
|---|---|---|
| Series rule | A = A1 x A2 x ... x An | Three 99.9% parts in series: 99.7% |
| Parallel rule | A = 1 - (1 - A1) x (1 - A2) x ... | Two 99% parts in parallel: 99.99% |
| App tier above | 1 - 0.001 x 0.001 | 99.9999% |
| Whole path | 0.9999 x 0.999999 x 0.9995 | about 99.94%, roughly 5 hours down per year |
| Add a DB standby with failover | DB becomes 1 - 0.0005^2, in theory | Path is now limited by the LB: about 99.99% |
- Redundancy only helps if failures are independent: same rack, same AZ, same bad deploy or same poisoned request breaks the assumption.
- Failover is not instant or perfect; detection time and failed failovers count as downtime.
- Every hard dependency you add in series lowers your ceiling. Make dependencies soft (cache, default, degrade) where you can.
Failure mode: correlated failure. Most large outages are a config push, a bad deploy or an overload cascade that hits all replicas at once, not independent hardware faults.
Reliability vs availability vs durability
| Property | Question it answers | Measured by | Bought with |
|---|---|---|---|
| Availability | Does it answer right now? | Successful requests / total; nines | Redundancy, failover, load shedding |
| Reliability | Does it do the correct thing over time, even under faults? | Error rate, MTBF, correctness of results | Testing, idempotency, safe deploys, fault isolation |
| Durability | Once acknowledged, is the data ever lost? | Probability of loss per year (for example "eleven nines" for object stores) | Replication, fsync, checksums, backups, restore drills |
- Available but not reliable: answers every request, sometimes with wrong or stale data.
- Durable but not available: data is safe on disk, the service is down.
- Related terms: fault tolerance (keeps working when a component fails), resilience (recovers quickly), MTTR (mean time to recover). Availability is about MTBF / (MTBF + MTTR), so cutting recovery time is usually cheaper than preventing every failure.
Consistency models as a ladder
Stronger models are easier to program against and cost more latency and availability. Pick the weakest model that keeps the product correct, per operation, not per system.
| Model | Guarantee in one line | When it matters |
|---|---|---|
| Linearizable | Behaves like a single copy; every operation takes effect at one point in real time, and later reads see it | Locks, leader election, unique constraints, balances, inventory decrement |
| Sequential | All clients see the same single order of operations, but that order may lag real time | Replicated logs, state machines fed from an ordered log |
| Causal | Operations that are causally related are seen in order by everyone; concurrent ones may differ | Comment threads (reply never appears before its parent), collaborative apps |
| Read-your-writes | A client always sees its own earlier writes | Profile edit then reload; post then view own timeline |
| Monotonic reads | A client never sees time go backwards across successive reads | Pagination and refresh across replicas with different lag |
| Eventual | If writes stop, all replicas converge; no bound on when, no ordering promise meanwhile | Like counts, view counts, DNS, caches, search indexes |
- "Strong consistency" in interviews nearly always means linearizable reads and writes on one object. Serializability is a different axis: isolation of multi-object transactions (see Data Systems).
- Cheap ways to get session guarantees on an eventually consistent store: read from the leader for a short window after a write, sticky routing to one replica, or pass a version token and wait for the replica to catch up.
CAP and PACELC
CAP, stated precisely: when a network partition separates replicas, a system must choose between consistency (C, meaning linearizability) and availability (A, meaning every non-failed node answers). It is not "pick two of three": partitions are not optional, so the real choice is C or A during a partition. PACELC adds the everyday case: else (E), with no partition, you still trade latency (L) against consistency (C).
| Class | Behaviour | Commonly cited examples (depends on configuration) | Fits |
|---|---|---|---|
| PA / EL | Always writable, low latency, reconcile later | Dynamo-style stores such as Cassandra with low consistency levels | Carts, feeds, telemetry, session data |
| PC / EC | Consistent always; pays latency, loses availability on the minority side | Consensus-based stores: Spanner-style databases, etcd, ZooKeeper | Money, inventory, metadata, coordination |
| PC / EL | Fast when healthy, refuses rather than diverges when partitioned | Less common; some single-leader setups with async replicas and strict failover | Systems that prefer downtime to split brain |
| PA / EC | Consistent when healthy, stays up by relaxing consistency under partition | Some configurable stores | Rarely a deliberate target |
- Many stores are tunable per request (consistency level, quorum size), so classify the configuration, not the product.
- CAP's "available" is stricter than SLA availability, and its "consistent" is unrelated to the C in ACID.
Interview tip: do not say "this is a CP system" and stop. Say which operation needs which: "placing an order is CP; browsing the catalogue is AP and may be seconds stale".
ACID vs BASE, strong vs eventual
| ACID | BASE | |
|---|---|---|
| Stands for | Atomicity, Consistency (invariants hold), Isolation, Durability | Basically Available, Soft state, Eventually consistent |
| Promise | A transaction is all-or-nothing and invariants are never visibly broken | The system stays up and converges; the application handles intermediate states |
| Cost | Coordination: locks or MVCC, sync commit; hard across shards or regions | Conflict resolution, compensation, idempotency, harder reasoning |
| Typical home | Relational and distributed SQL databases | Dynamo-style key-value and wide-column stores, caches, search indexes |
| Data | Needs | Why |
|---|---|---|
| Bank balance, wallet, ledger | Strong (linearizable, transactional) | A stale read allows a double spend; invariants must hold |
| Inventory, seat booking | Strong on the decrement / reservation | Overselling is a real-world cost |
| Unique username, idempotency key | Strong | Two winners is a correctness bug |
| Like count, view count, follower count | Eventual | Off by a few for a few seconds is invisible |
| Feed, search results, recommendations | Eventual | Freshness in seconds is fine; availability and latency matter more |
| Own profile after editing | Read-your-writes | Others may see it late; the author must not |
Read-heavy vs write-heavy
The read:write ratio is the first number that shapes a design. Read-heavy systems move work to write time (precompute, cache, replicate). Write-heavy systems move work off the write path (append, buffer, batch, partition).
| Read-heavy (for example 100:1) | Write-heavy (for example 1:10) | |
|---|---|---|
| Examples | Feeds, product pages, URL redirects, profiles | Metrics, logs, IoT, click streams, chat messages, location pings |
| Main tools | Cache, CDN, read replicas, denormalised read models, indexes | Append-only log, queue buffering, batching, sharding, LSM storage |
| Storage bias | B-tree, many indexes | LSM-tree, few indexes |
| You pay | Staleness, invalidation complexity, slower writes | Read-time work (merge, aggregate), eventual visibility |
| Watch for | Cache stampede, replica lag, hot keys | Backpressure, hot partitions, compaction stalls |
Push vs pull, sync vs async, stateful vs stateless
| Axis | Option | Gain | Pay | Use when |
|---|---|---|---|---|
| Delivery | Push (fan-out on write, server-sent) | Fast reads, fresh data | Write amplification, wasted work for inactive receivers | Few receivers per event, low latency required |
| Pull (fan-out on read, polling) | Cheap writes, receiver controls rate | Slow or heavy reads, staleness up to the poll interval | Huge fan-out (celebrity), infrequent readers | |
| Coupling in time | Synchronous (request-response) | Simple, immediate result and error | Caller blocked; availabilities multiply; cascading failure | User needs the answer now; short call chain |
| Asynchronous (queue, event) | Absorbs spikes, isolates failures, retries for free | Eventual result, duplicates, ordering, harder debugging | Slow or unreliable work; spikes; multiple consumers | |
| Server state | Stateless | Any instance serves any request; trivial scaling and failover | Every request pays a state lookup | Default for request-serving tiers |
| Stateful | Data locality, no lookup, long-lived connections | Sticky routing, rebalancing, careful deploys, replication of state | WebSocket gateways, databases, stream processors, game servers |
Trade-off: hybrids are normal. Feeds push to ordinary users and pull for celebrity accounts; APIs accept synchronously (202 plus a status URL) and process asynchronously.
Single-leader vs multi-leader vs leaderless
Where writes are accepted decides whether conflicts can exist. Details (replication lag, quorums, conflict resolution) are in Data Systems and Distributed Systems Patterns.
| Single-leader | Multi-leader | Leaderless | |
|---|---|---|---|
| Writes go to | One node | Any leader, usually one per region | Any replica; W of N must acknowledge |
| Write conflicts | None | Yes; need resolution (last-write-wins, CRDT, app merge) | Yes; versions, read repair, anti-entropy |
| Consistency available | Strong from the leader; stale from async followers | Eventual across leaders | Tunable; R + W > N gives overlap, not full linearizability |
| Write availability | Lost until failover completes | Survives loss of a leader or region | Survives node loss with no failover step |
| Write scaling | One node (shard to go further) | Per region, not per key | Scales with nodes |
| Main risk | Failover: lost async writes, split brain | Conflict handling bugs | Stale reads, sloppy quorums, LWW data loss |
| Typical use | Most OLTP: PostgreSQL, MySQL, MongoDB replica sets | Multi-region writes, offline-capable clients | Dynamo-style stores: Cassandra, Riak |
Interview tip: default to single-leader with replicas. Move away only when you can name the requirement: writes in several regions, or write availability through node loss.
Scaling a design step by step
Part 5 of 5 · from one server to millions of users
The classic evolution. Each step is taken because a specific limit was hit, not in advance. In an interview, start at the step your numbers justify and name the trigger for the next one.
Steps and their triggers
| Step | Change | Trigger | New problem it brings |
|---|---|---|---|
| 1 | Single server | Start here | One failure domain; app and DB compete for resources |
| 2 | Separate database server | CPU / memory contention; need to scale tiers independently | Network hop per query |
| 3 | Load balancer + stateless app tier | App CPU saturated; need zero-downtime deploys and redundancy | Sessions must move out of the process |
| 4 | Read replicas | DB read load saturates the primary; need failover target | Replication lag, stale reads |
| 5 | Cache | Same hot rows read repeatedly; DB latency dominates p99 | Invalidation, stampedes, hot keys |
| 6 | CDN | Static and media bytes dominate bandwidth; users are far away | Purging, cache-control correctness |
| 7 | Queue + workers | Slow or spiky work on the request path | Duplicates, ordering, monitoring lag |
| 8 | Sharding | Write volume or dataset exceeds one primary | Cross-shard queries and transactions, rebalancing, hot shards |
| 9 | Multi-region | Global latency, region-level disaster recovery, data residency | Cross-region replication, conflicts, cost |
Interview tip: steps 3 to 7 are nearly free of regret and can be drawn immediately at any real scale. Steps 8 and 9 are expensive; justify them with a number.
1 and 2: one box, then a separate database
- Good for: a few hundred requests per second with vertical scaling. Many products never need more than step 2 plus backups.
- Trigger to move on: the app server is CPU-bound, or any deploy or crash is an outage.
3: load balancer and stateless app tier
- Gain: horizontal app capacity, rolling deploys, health-checked failover. Run the load balancer itself as a redundant pair.
- Trigger to move on: the database is now the bottleneck, usually on reads first.
4: read replicas
- Gain: read capacity and a promotion target if the primary fails.
- Failure mode: read-after-write returns stale data. Route a user's reads to the primary briefly after their own write.
- Trigger to move on: the same queries repeat, or replicas keep being added for a few hot rows.
5: cache
- Gain: sub-millisecond reads for hot data and an order of magnitude less database load.
- Pay: staleness and invalidation. The database must still survive a cold cache. Patterns and failure modes are in Building Blocks.
- Trigger to move on: bandwidth and latency for static assets and media.
6: CDN
- Gain: lower latency worldwide, origin offload, absorption of traffic spikes.
- Media goes to object storage, not app-server disks; uploads use pre-signed URLs so bytes bypass the app tier.
- Trigger to move on: slow or bursty work (email, thumbnails, fan-out) is making requests slow.
7: queue and workers
- Gain: fast responses, spike absorption, independent scaling and failure isolation of slow work.
- Pay: at-least-once delivery means handlers must be idempotent.
- Trigger to move on: the primary database cannot keep up with writes, or the dataset no longer fits one node comfortably.
8: sharding
- Before sharding: bigger primary, archive cold data, split by function (move a table or service to its own database).
- Choose the shard key from the dominant access pattern; queries that do not include it become scatter-gather.
- Trigger to move on: users on other continents, or a requirement to survive the loss of a region.
9: multi-region
| Mode | Writes | Gain | Pay |
|---|---|---|---|
| Active-passive | One region; the other is a warm standby | Simple; disaster recovery | Failover time; possible loss of the last async writes; far users stay slow on writes |
| Active-active, partitioned by user home region | Each user writes in one region | Low latency, no conflicts for most data | Routing, user migration, global data still needs a plan |
| Active-active, multi-leader | Any region for any data | Best write latency and availability | Conflict resolution everywhere |
Cheat sheet
| Topic | Remember |
|---|---|
| Framework | Clarify 5, estimate 4, API 4, data model 4, high-level 10, deep dives 12, bottlenecks 4, wrap-up 2 (minutes) |
| Requirements | Functional, non-functional, out of scope. Ask scale, latency, consistency; assume the rest out loud |
| Estimation | Day = 10^5 s. 1M/day = about 10/s. 1M x 1 KB = 1 GB. Peak = 2x to 3x. 5 years = about 2,000 days |
| Latency | RAM 100 ns, SSD 100 us, datacentre hop 0.5 ms, disk seek 10 ms, cross-continent 100 to 150 ms |
| One node (approx.) | App 1k rps, SQL 10k reads/s and 1k to 5k writes/s, Redis-like 100k ops/s, Kafka-like 100k+ msgs/s |
| Nines | 99.9% = 8.8 h/year, 99.99% = 53 min/year, 99.999% = 5 min/year |
| Availability maths | Series multiply availabilities; parallel multiply failure probabilities; correlated failures break both |
| Percentiles | Quote p99. Fan-out of 100 turns backend p99 into a 63% chance of a slow request |
| Consistency ladder | Linearizable, sequential, causal, read-your-writes / monotonic reads, eventual. Weakest that is correct, per operation |
| CAP / PACELC | During a partition choose C or A; otherwise choose latency or consistency |
| Strong vs eventual | Money, inventory, uniqueness: strong. Counts, feeds, search: eventual. Own edits: read-your-writes |
| Read-heavy | Cache, CDN, replicas, precomputed read models |
| Write-heavy | Append-only log, queue, batching, sharding, LSM storage |
| Replication | Single-leader by default; multi-leader for multi-region writes; leaderless for write availability |
| Scaling order | Separate DB, LB + stateless apps, replicas, cache, CDN, queue, sharding, multi-region; each with a trigger |
Next in the path: Building Blocks.