System design, part 7 of 12
Reliability and Operations
A design that works on a whiteboard is not yet a design that runs. This page covers what separates the two: how reliability is defined and measured, how a system is built to survive failures, how it is observed, shipped, sized and operated, and how to weave all of it into an interview answer. Interviewers at senior level listen for this explicitly.
Defining reliable
Part 1 of 7 · targets you can measure
"Highly available" means nothing until it has a number, a measurement and a consequence. Start every reliability discussion here.
- SLI, SLO, SLA: indicator, objective, contract
- Choosing good SLIs: availability, latency, freshness, correctness
- Error budgets: how they drive release decisions
- Availability arithmetic: nines, series, parallel
- RTO and RPO: how long down, how much lost
SLI, SLO, SLA
| What it is | Example | Owned by | |
|---|---|---|---|
| SLI (indicator) | A measured ratio: good events / valid events | Share of checkout requests that return non-5xx in under 300 ms | Engineering |
| SLO (objective) | Target for an SLI over a window | 99.9% of checkout requests good, over a rolling 28 days | Engineering plus product |
| SLA (agreement) | Contract with a penalty if missed | 99.5% monthly uptime or 10% service credit | Business and legal |
- Measure the SLI as close to the user as you can: load balancer logs beat server metrics, client telemetry beats both.
- Never promise 100%. Every dependency, network and client you do not control is below 100%, and the last nine costs the most.
Choosing good SLIs
Pick 2 to 4 SLIs per user journey, chosen by what the user would complain about, not by what is easy to scrape.
| SLI type | Good event | Fits | Pitfall |
|---|---|---|---|
| Availability | Request served without a server error | Request/response APIs | Counting 4xx as bad (client bugs) or ignoring requests that never reached you |
| Latency | Served faster than a threshold, at a percentile (p50, p99) | Interactive paths | Using the mean, which hides the tail; averaging percentiles across hosts |
| Freshness | Data age below a bound (feed updated within 60 s) | Pipelines, caches, replicas, search indexes | Measuring job success instead of data age |
| Correctness | Output matches a known answer (probes, reconciliation) | Billing, ledgers, ML outputs | Only checked by users, months later |
| Durability | Written data still readable | Storage systems | Never tested until a restore is needed |
Senior signal: express latency SLOs as "99% of requests under 300 ms", not "average 100 ms". A p99 target on a page that fans out to 20 backends means each backend needs a much tighter tail.
Error budgets
Error budget = 1 - SLO. A 99.9% SLO over 30 days allows 0.1% bad requests, about 43 minutes of full outage. It turns "reliability vs velocity" into a shared, numeric rule.
- Unspent budget is a signal too: the SLO may be too loose, or the team too cautious. Spend it on risky migrations and chaos tests.
- Budget consumed by a dependency still counts for the user; it is an input to the dependency's own SLO discussion.
Availability arithmetic
| Availability | Downtime / year | / 30-day month | / day | Realistic for |
|---|---|---|---|---|
| 99% (two nines) | 3.65 days | 7.2 h | 14.4 min | Internal batch tools |
| 99.9% | 8.76 h | 43.2 min | 1.44 min | Most product services in one region |
| 99.95% | 4.38 h | 21.6 min | 43 s | Typical managed cloud service SLA |
| 99.99% | 52.6 min | 4.3 min | 8.6 s | Payments, auth; needs multi-zone and automated failover |
| 99.999% | 5.3 min | 26 s | 0.86 s | Multi-region, no human in the recovery loop |
Series: every hop must work
Parallel: any one copy is enough
- Series: A = A1 × A2 × ... A chain of ten 99.9% dependencies is about 99.0%: three nines decay to two.
- Parallel: A = 1 - (1 - a)n, but only for independent failures with working failover. Shared config, a shared deploy or one zone breaks independence.
Interview tip: do this multiplication out loud once. It justifies why you add replicas, why you remove synchronous hops, and why a 99.99% target needs the database tier to be redundant too.
RTO and RPO
RPO (recovery point objective): how much recent data you can afford to lose. RTO (recovery time objective): how long you can be down. Both are business inputs; they pick the replication and DR strategy.
- RPO near zero needs synchronous replication (pay write latency) or a replicated log; RPO of minutes allows async replication or continuous log shipping.
- RTO under minutes needs automated detection and failover; anything with "a human runs the runbook" is 15+ minutes in practice.
Designing for failure
Part 2 of 7 · assume every box dies
Every component fails; the design decides whether users notice. Remove single points of failure, decide how failover works, and decide what to switch off when things go wrong.
- Redundancy levels: instance, zone, region
- Single points of failure: how to find them
- Failover: active-passive and active-active
- Multi-region designs: and the replication behind each
- Disaster recovery strategies: cost vs RTO and RPO
- Backups that are tested
- Graceful degradation and load shedding
- Critical vs optional dependencies
Redundancy levels
| Level | Survives | Cost | Latency impact |
|---|---|---|---|
| Instance (N+1, N+2) | Host crash, bad deploy on one node | Low: one or two spare instances | None |
| Zone (multi-AZ) | Power, network or cooling loss in one data centre | Moderate: cross-zone traffic is billed; capacity for N-1 zones | About 1 to 2 ms between zones; sync replication is affordable |
| Region | Regional outage, control-plane failure, disaster | High: duplicate stack, egress, operational complexity | Tens to 100+ ms between regions; sync replication hurts |
Failure mode: three zones, but peak needs all three of them busy. Losing one zone then overloads the other two and the outage spreads. Provision so N-1 zones can carry peak.
Single points of failure
Walk the request path on your diagram and ask, for every box and arrow: "if exactly this dies, does the request fail?" Any yes without a mitigation is a SPOF.
- Easy to miss: DNS, certificates (expiry), the secrets store, the auth service, the CI system you need to deploy a fix, a single NAT gateway, one on-call person.
- Also a SPOF: a shared thing replicated everywhere, such as one global config push or one bad deploy to all regions at once.
Failover: active-passive and active-active
Active-passive
Active-active
| Active-passive | Active-active | |
|---|---|---|
| Failover time | Seconds to minutes (detect, promote, DNS TTL, cold caches) | Near zero; route around the dead site |
| Cost | Standby capacity sits idle (can be smaller if warm) | Each side needs headroom to carry 100% alone |
| Data | One writer; simple | Multiple writers need conflict handling or partitioned ownership |
| Risk | Standby rots untested; split brain if both think they are primary | Hidden cross-site dependencies; capacity mistakes cause cascades |
Failure mode: split brain. Use fencing (a lease or epoch number the old primary must hold to write) so a partitioned old primary cannot keep accepting writes. Consensus and leases are covered in Distributed Systems Patterns.
Multi-region designs
Go multi-region for latency to global users, for regional disaster survival, or for data residency. Each shape is decided by where writes go.
| Shape | Writes go to | Replication choice | Gain | Pay |
|---|---|---|---|---|
| Active-passive region | Primary region only | Async to standby region | Simple; DR | RPO = replication lag; RTO = failover drill |
| Read-local, write-global | One home region | Single leader, async read replicas per region | Fast reads everywhere | Slow writes for remote users; stale reads; use read-your-writes routing |
| Partitioned by user (geo-homing) | The user's home region | Each partition has a leader in its home region | Fast local writes; residency | Cross-region users and global queries are hard; re-homing |
| Active-active multi-writer | Any region | Multi-leader or leaderless, conflict resolution (LWW, CRDTs, app merge) | Local writes, survive region loss | Conflicts, anomalies; only for data that merges well |
| Global consensus | Any region, committed by quorum | Synchronous quorum across 3+ regions (Spanner-style) | Strong consistency, RPO 0 | Write latency of a cross-region quorum, 50 to 200 ms |
| Follow-the-sun | Region that owns the current time window | Ownership moves on a schedule; async catch-up | Writes local to the active workforce | Handover windows; mostly for internal tools and ops |
Senior signal: split the data. Keep money and inventory single-writer or consensus-backed, and let carts, likes and presence be active-active with merges. One replication model for the whole system is rarely right. Replication and conflict mechanics are in Data Systems.
Disaster recovery strategies
| Strategy | What runs in the DR region | RTO (approx.) | RPO (approx.) | Cost |
|---|---|---|---|---|
| Backup and restore | Nothing; backups copied there | Hours to a day | Hours (last backup); minutes with continuous log backup | $ |
| Pilot light | Data replicated; compute off or minimal | Tens of minutes to hours | Seconds to minutes | $$ |
| Warm standby | Full stack, scaled down, serving no users | Minutes | Seconds | $$$ |
| Multi-site active-active | Full stack, serving users | Near zero | Near zero to seconds | $$$$ |
- Pick per data set and per service, from the business RTO / RPO, not one strategy for the whole company.
- Infrastructure as code is what makes pilot light and warm standby realistic: the DR stack is the same templates with different sizes.
Backups that are actually tested
A backup you have never restored is a hope. Replication is not a backup: it copies a bad DELETE to every replica in milliseconds.
- 3-2-1: three copies, two media or services, one off-site (another region or account).
- Immutable / separate account: object lock or a separate account so a compromised credential or ransomware cannot delete them.
- Point-in-time recovery: snapshot plus write-ahead log lets you restore to just before the bad change.
- Restore drills on a schedule: automated restore into a scratch environment, row-count and checksum checks, and the measured duration recorded as your real RTO.
Graceful degradation and load shedding
When capacity or a dependency is lost, serve a smaller product instead of no product, and reject excess work early and cheaply instead of slowing down for everyone.
- Degradation examples: show cached prices with a "may be out of date" note, hide recommendations, disable search filters, switch to read-only mode.
- Shedding: bound queues and concurrency; drop work whose client has already timed out; prefer checkout traffic over crawlers and batch jobs.
- Pre-build the switches (feature flags, kill switches) and test them; you will not write them safely during an incident.
Critical vs optional dependencies
Classify every dependency of every request. The answer decides timeouts, fallbacks and whether its outage is your outage.
| Class | If it fails | Treatment |
|---|---|---|
| Critical | The request fails | Redundant, SLO at least as tight as yours, retries with budget, circuit breaker that fails fast |
| Optional (soft) | Feature hidden or defaulted | Tight timeout, fallback value, never retried at length |
| Asynchronous | Work delayed | Queue with retries and a dead-letter queue |
Circuit breakers, retries with backoff and jitter, timeouts, bulkheads and backpressure are covered in Distributed Systems Patterns.
Observability
Part 3 of 7 · knowing what it is doing
Monitoring tells you that something is wrong; observability lets you ask why, including questions you did not plan for. Both are built from three signals.
- Metrics, logs, traces: what each answers
- Metric types: and why percentiles need histograms
- RED, USE and the golden signals
- Distributed tracing: trace, span, propagation
- Structured logs and correlation ids
- Alerting: symptoms, fatigue, burn rate
- Dashboards worth having
Metrics, logs, traces
| Signal | Answers | Shape | Cost driver | Weak at |
|---|---|---|---|---|
| Metrics | Is something wrong, and how much? Trends, alerts | Numeric time series with a few labels | Label cardinality (never label by user id) | Explaining one specific request |
| Logs | What exactly happened in this component? | Timestamped events, ideally structured | Volume: bytes ingested and retained | Cross-service flow; aggregation is expensive |
| Traces | Where did the time go across services for this request? | Tree of timed spans per request | Span volume; usually sampled | Rare events if sampled away |
The workflow: an alert fires on a metric, an exemplar or filter leads to a slow trace, the trace id leads to the logs of the failing span. Link them with shared ids.
Metric types
| Type | Behaviour | Example | Query as |
|---|---|---|---|
| Counter | Only goes up (resets on restart) | http_requests_total{status="500"} | Rate per second |
| Gauge | Goes up and down; a current value | Queue depth, memory in use, connections | Current value, max, min |
| Histogram | Counts observations into buckets | Request duration buckets 10, 50, 100, 300, 1000 ms | Percentiles estimated from bucket rates |
- You cannot average percentiles: the mean of 20 hosts' p99 is not the fleet p99. Histograms can be summed across hosts, then the percentile computed, which is why latency SLOs need them.
- Choose bucket edges around your SLO threshold (for example one at exactly 300 ms) so "share under 300 ms" is exact.
RED, USE and the four golden signals
| Method | For | Measure |
|---|---|---|
| RED | Request-driven services | Rate, Errors, Duration |
| USE | Resources: CPU, memory, disk, network, pools, queues | Utilisation, Saturation (queued work), Errors |
| Four golden signals (Google SRE book) | Any user-facing system | Latency, Traffic, Errors, Saturation |
RED tells you users are hurting; USE tells you which resource is why. Separate the latency of successful and failed requests: fast errors make a broken service look quick.
Distributed tracing
A trace is one request's journey; a span is one timed operation in it with a parent span. Context propagation passes trace id and parent span id across process boundaries, usually in the W3C traceparent header or in message headers for queues.
- Sampling: head sampling (decide at the start, for example 1%) is cheap but misses rare errors; tail sampling (decide after the trace ends) keeps all slow and failed traces at the cost of buffering.
- Propagation breaks silently at thread pools, queues and third-party SDKs; check that async consumers continue the trace.
- OpenTelemetry is the common vendor-neutral standard for emitting all three signals.
Structured logs and correlation ids
Log events as key-value records, not sentences, so they can be filtered and aggregated. Every line carries the request's correlation id (often the trace id).
- Generate the id at the edge, return it to the client (support tickets can quote it), propagate it everywhere including queue messages.
- Levels with meaning: error = someone should look; warn = degraded but handled; info = business events; debug off in production or sampled.
- No secrets or raw personal data in logs; hash or omit. Logs are often kept longer and seen by more people than the database.
Python specifics (the logging module, structured output, debugging) are in Debugging and Logging.
Alerting
| Do | Avoid |
|---|---|
| Page on symptoms users feel: SLO burn, error rate, latency | Paging on causes: CPU 80%, one pod restarted, disk 70% |
| Every page is urgent, actionable and has a runbook link | Alerts nobody acts on (they train people to ignore pages) |
| Causes go to tickets or dashboards | Duplicate alerts for one incident from every layer |
| Review alert volume weekly; delete or fix noisy ones | Static thresholds tuned once and forgotten |
Burn-rate alerts: burn rate is how fast you are spending the error budget relative to plan (1 = exactly used up at the end of the window). A common setup pages when the 1-hour burn rate exceeds about 14 (2% of a 30-day budget gone in an hour, confirmed over a short window too) and opens a ticket at a slower burn over 6 hours or more.
Failure mode: alert fatigue. A team paged ten times a night stops reading pages, and the real one is missed. Fewer, symptom-based pages are a reliability feature.
Dashboards worth having
- Service overview: RED per endpoint, SLO and remaining error budget, deploy markers on every graph.
- Dependencies: latency and error rate of each downstream call, circuit breaker state.
- Resources: USE for CPU, memory, connection pools, thread pools, queue depth and consumer lag.
- Business: orders per minute, sign-ups, payments succeeded. Often the first to show a silent failure.
A dashboard should answer one question in ten seconds at 3 a.m. Dozens of unlabelled graphs are not observability.
Shipping safely
Part 4 of 7 · change without outages
Most outages start with a change. Safe delivery limits how many users a bad change reaches and how quickly it can be undone.
- Deployment strategies: recreate to shadow
- Feature flags and progressive delivery
- Zero-downtime schema migrations: expand, migrate, contract
- Backward-compatible changes
- Rollback vs roll-forward
- CI/CD pipeline: and infrastructure as code
Deployment strategies
Rolling
Blue-green
Canary
Shadow (dark launch)
| Strategy | User risk | Extra cost | Rollback speed | Watch out for |
|---|---|---|---|---|
| Recreate | Downtime for all | None | Slow: redeploy old version | Only for dev or tolerated maintenance windows |
| Rolling | Grows with each batch | Little (surge capacity) | Medium: roll back batch by batch | Mixed versions; must be compatible |
| Blue-green | All at once on switch | Up to 2x during release | Seconds: flip the router | Shared DB and caches; sessions on switch |
| Canary | Small slice first | Small | Fast: drain the canary | Needs good metrics and enough canary traffic to be significant |
| Shadow | None (responses discarded) | Up to 2x for mirrored paths | Nothing to roll back | Side effects; only validates, never releases |
Feature flags and progressive delivery
Flags separate deploy (code is on servers) from release (users see it). Progressive delivery widens the audience in steps, gated by metrics.
- Kinds: release flags (short-lived), ops kill switches (permanent), experiment flags (A/B), permission flags (plans, tenants).
- Trade-off: every flag doubles a code path. Give release flags an owner and an expiry date, and delete them.
- Flag evaluation must not be a critical dependency: cache flags locally, with safe defaults if the flag service is down.
Zero-downtime schema migrations
Old and new code run together during any rolling deploy, so every schema state must work with both. Example: rename name to full_name.
- Backfill in small batches with throttling; one giant
UPDATElocks rows and floods replication. - Watch for locking DDL: adding a column with a default, adding an index or a NOT NULL constraint can lock a big table on some engines and versions; use online schema change tools or concurrent index builds.
- Contract only after every service, job and old mobile client has stopped reading the old column.
Backward-compatible changes
| Safe | Breaking |
|---|---|
| Add an optional field or endpoint | Remove or rename a field |
| Accept a new enum value on read (ignore unknowns) | Send a new enum value to old readers that reject it |
| Widen what you accept | Narrow validation, change a type or units |
| New message version alongside the old | Change the meaning of an existing field |
Rule: deploy readers before writers. Consumers must tolerate the new shape before any producer emits it. For events, a schema registry with compatibility checks enforces this. API versioning is in API and Communication Design.
Rollback vs roll-forward
- Roll back first when the last change is the likely cause. Mitigate, then investigate. Rollback must be one command or one flag flip and rehearsed.
- Roll forward when rollback is unsafe: a data migration already ran, a new format was written that old code cannot read, or the fix is tiny and well understood.
- Design so rollback is always possible: expand-contract migrations, readers before writers, flags around risky paths.
Senior signal: "every deploy must be reversible for at least one release" is a design constraint, not an ops detail. It shapes migrations and message formats.
CI/CD pipeline
- Deploy regions or cells one wave at a time with bake time between, so a bad release never reaches everyone at once.
- Infrastructure as code (Terraform, CloudFormation and similar): servers, networks and config are versioned, reviewed and reproducible, which is what makes DR stacks and identical environments possible.
Capacity and performance
Part 5 of 7 · enough, but not too much
Turn traffic estimates into machines, keep headroom for peaks and failures, and know where the system will break first.
- Capacity planning: QPS to instance counts
- Autoscaling: reactive, predictive, stateful
- Load testing types
- Finding bottlenecks
- Queueing intuition: why 100% is a failure
- Cost as a design constraint
Capacity planning
Worked example: 10 M daily users, 20 requests each, peak 3x average; one instance handles 800 QPS within the latency SLO in a load test and runs at 60% of that; 3 zones, must survive losing one.
- 15 instances carry peak; 24 are deployed so any two zones can carry it. Zone redundancy costs 60% more here, and that is the price of the SLO.
- Repeat for each tier: the database, cache memory and queue throughput usually run out before stateless compute.
- Plan for growth: forecast 6 to 12 months ahead for anything with a long lead time (database shards, reserved capacity).
Autoscaling
| Kind | How | Good for | Weak at |
|---|---|---|---|
| Reactive (target tracking) | Keep a metric near a target, for example 60% CPU or 400 RPS per instance | Gradual changes | Sudden spikes faster than boot time |
| Scheduled / predictive | Scale ahead of known patterns (daily peak, a sale, a match) | Predictable load | Surprises |
| Queue-based | Workers sized by backlog / acceptable drain time | Async consumers | Partitioned queues cap useful consumers |
- Cooldowns and asymmetric thresholds stop flapping. Keep a minimum that survives a zone loss.
- The stateful tier is the hard part: adding a database node means moving data (resharding, rebalancing) which takes hours and adds load exactly when you are busiest. Plan stateful capacity ahead; autoscale the stateless tiers.
- Scaling out the app can overload the database (more connections); use pooling and cap the fleet.
Load testing types
| Type | Shape of load | Finds |
|---|---|---|
| Load | Expected peak, held | Whether you meet the SLO at planned capacity |
| Stress | Increase until it breaks | The breaking point and how it fails (gracefully or by collapse) |
| Soak | Normal load for hours or days | Leaks: memory, connections, file handles, growing queues |
| Spike | Sudden jump, for example 10x in a minute | Autoscaling lag, cold caches, thundering herds |
Use production-like data sizes and realistic request mixes; a test against an empty database or a single hot key measures the wrong thing.
Finding bottlenecks
- Start from the slowest span in traces, then apply USE to that component: which resource is saturated?
- Usual suspects, in rough order: database (slow queries, missing indexes, lock contention, connection pool), hot keys or partitions, synchronous calls to slow dependencies, thread or connection pool limits, GC pauses, network egress.
- Profile under load (CPU flame graphs, allocation profiles); the code that is slow at 1 QPS is often not the one slow at 5,000.
- Fix the biggest one, measure again. The bottleneck moves; optimising a non-bottleneck changes nothing.
Queueing intuition
In a simple queue model (M/M/1), mean response time is service time / (1 - utilisation). Latency stays flat until about 70% and then climbs steeply.
| Utilisation | 50% | 70% | 80% | 90% | 95% | 99% |
|---|---|---|---|---|---|---|
| Response time vs idle | 2x | 3.3x | 5x | 10x | 20x | 100x |
- At 100% utilisation the queue grows without bound: every request times out, clients retry, and load rises further. That is why it is a failure, not efficiency.
- This is why targets sit at 50 to 70%, why queues need bounds, and why load shedding beats queueing during overload.
- Little's law: items in system = arrival rate × time in system. At 2,000 QPS and 50 ms, about 100 requests are in flight; size pools from that.
Cost as a design constraint
| Cost | Grows with | Levers |
|---|---|---|
| Compute | Peak load, headroom, idle standby | Right-size instances, autoscale, reserved or committed capacity for the base, spot for batch |
| Storage | Data volume times replicas times retention | Tiered storage (hot SSD, warm, cold object store, archive), TTLs, compression |
| Egress | Bytes leaving a region or the cloud; cross-zone traffic | CDN, compression, keep chatty services in one zone, avoid cross-region fan-out |
| Observability | Log volume, metric cardinality, trace volume | Sampling, retention tiers, drop debug logs |
Senior signal: mention cost when choosing a reliability level. "Active-active in three regions roughly triples the infrastructure bill; for this product a warm standby meets the RTO" is the kind of trade-off interviewers want to hear.
Operating it
Part 6 of 7 · people and process
Reliability is also a practice: how incidents are handled, learned from and prevented.
- Incident response: detect to review
- On-call and runbooks
- Blameless postmortems
- Chaos engineering and game days
- Change management: the top cause of incidents
- Production readiness checklist
Incident response
- Roles: incident commander (coordinates, decides), operations lead (hands on keyboard), communications lead (status page, stakeholders). Small incidents may combine them.
- Severity levels agreed in advance (for example SEV1: major user impact, all hands) decide who is paged and how often to update.
- Keep a timeline as you go in the incident channel; it becomes the postmortem.
- Key measures: time to detect, time to mitigate. MTTR matters more than MTBF for most systems.
On-call and runbooks
- Rotation of at least 5 to 8 people per schedule, a primary and a secondary, a clear escalation path, and a handover note each shift.
- Measure load: pages per shift and night pages. A sustained couple of incidents per shift means the system needs engineering, not more heroes.
- Runbooks: linked from each alert; state what the alert means, how to confirm user impact, safe mitigations (commands, dashboards, flags), and when to escalate. Automate steps that are run often.
- The team that builds a service is on call for it, or at least owns its alerts; it closes the feedback loop.
Blameless postmortems
Blameless means asking why the system let a reasonable person make the mistake, not who made it. People hide information when they fear blame, and the real causes stay hidden.
| Section | Contains |
|---|---|
| Summary and impact | Who was affected, how much, how long; error budget spent |
| Timeline | Detection, key decisions, mitigation, resolution, with timestamps |
| Contributing factors | Usually several: the trigger, the latent bug, the missing guard, why detection was slow |
| What went well, what was lucky | Luck is a risk to remove |
| Action items | Specific, owned, dated, prioritised; tracked to done |
Failure mode: action items that say "be more careful". Good actions change the system: a guard rail, an automated check, an alert, a safer default.
Chaos engineering and game days
- Faults to inject: kill instances, add latency to a dependency, drop a zone, fill a disk, expire a certificate, exhaust a connection pool, fail DNS.
- Game days: scheduled exercises where a team runs a failure scenario (often a DR failover) end to end, including the humans and runbooks.
- Start in staging, then production during working hours with small scope and error budget to spare.
Change management
Changes (code deploys, config pushes, feature flags, infrastructure, certificate and DNS updates) are widely reported as the largest single trigger of production incidents. Treat every change, not only code, as a deploy.
- Config and flags go through review, validation, staged rollout and quick rollback, just like code. A global config push is the classic way to take down every region at once.
- Every change is recorded with who, what and when, and shown as a marker on dashboards, so "what changed?" is answered in seconds.
- Freeze or slow changes during peak events and when the error budget is spent.
Production readiness checklist
| Area | Ready means |
|---|---|
| SLOs | SLIs defined and measured; SLO and error budget policy agreed |
| Redundancy | No SPOF on the critical path; N-1 zone capacity; failover tested |
| Dependencies | Each classified; timeouts, retries with budget, circuit breakers, fallbacks |
| Observability | RED dashboards, structured logs with correlation ids, tracing, deploy markers |
| Alerting | Symptom-based pages with runbooks; owner and on-call rotation |
| Delivery | Automated pipeline, canary or staged rollout, one-step rollback |
| Data | Backups with a tested restore; RPO and RTO met in a drill; migrations expand-contract |
| Capacity | Load tested at expected peak plus headroom; autoscaling limits; quotas raised |
| Overload | Rate limits, load shedding, bounded queues |
| Security | Secrets managed, least privilege, certificate expiry monitored |
In the interview
Part 7 of 7 · saying it out loud
Reliability is not a separate section at the end of an answer; it threads through it. These chapters show how, and summarise the page.
Weaving reliability into an HLD answer
| Phase of the answer | What to say |
|---|---|
| Requirements | Name SLOs up front: "99.9% for reads, 99.99% for payments, p99 under 200 ms; RPO zero for orders, minutes for analytics." |
| High-level diagram | Mark the single points of failure yourself before the interviewer does, and say how each is removed. |
| Deep dive | For each component: what happens when it dies, how it is detected, what the user sees. |
| Scale | Capacity arithmetic with headroom for a zone loss; where the stateful bottleneck is. |
| Wrap-up | How you would deploy it (canary, flags, expand-contract), observe it (SLO dashboards, burn-rate alerts) and recover it (DR strategy that matches the RTO). |
Senior signal: mid-level answers say "add replicas for availability". Senior answers say which failure each replica covers, what failover costs in time and data, and what the user sees meanwhile.
Component fails, then what?
A sample e-commerce architecture, then the table you should be able to produce for it.
| Component fails | User impact | Mitigation |
|---|---|---|
| One API instance | None; a few requests retried | Health checks remove it; N+2 capacity |
| A whole zone | Brief errors, then none | Two zones sized to carry peak; LB stops routing to the dead zone |
| Cache cluster | Slower pages; risk of DB overload | Replicated cache; request coalescing; shed low-priority reads; warm on recovery |
| Primary database | Writes fail for seconds to a minute | Automated failover to synchronous standby (RPO 0); fencing of the old primary |
| Queue | Emails and fulfilment delayed | Replicated queue; outbox table holds events until it returns |
| Workers | Backlog grows; no immediate user impact | Autoscale on lag; idempotent consumers; DLQ for poison messages |
| Payment provider | Cannot pay | Timeout and circuit breaker; second provider or "order accepted, payment pending" with retry |
| CDN | Slow or failing static assets | Origin can serve directly at reduced scale; DNS switch to a second CDN |
| Whole region | Outage until failover | Warm standby region; RTO minutes, RPO = async lag; rehearsed by game day |
| Bad deploy | Errors for the canary slice | Automatic canary abort and rollback |
Cheat sheet
| Topic | Remember |
|---|---|
| SLI / SLO / SLA | Measured ratio / internal target / contract with penalty; SLA looser than SLO |
| Error budget | 1 - SLO; 99.9% is about 43 min per 30 days; exhausted means freeze features |
| Nines | 99.9% = 8.8 h/yr; 99.99% = 53 min/yr; 99.999% = 5 min/yr |
| Composition | Series multiplies (ten 99.9% hops is about 99.0%); parallel 1 - (1 - a)^n if independent |
| RPO / RTO | Data you can lose / time you can be down; pick replication and DR from them |
| Redundancy | Instance, zone, region; N-1 zones must carry peak |
| Failover | Active-passive: cheap, slower, split-brain risk. Active-active: fast, 2x headroom, conflicts |
| DR ladder | Backup-restore (hours) to pilot light, warm standby (minutes), multi-site (near zero) |
| Backups | Replication is not a backup; 3-2-1; restore drills give the real RTO |
| Overload | Degrade optional features; shed low priority early; bound queues |
| Signals | Metrics: is it broken. Traces: where. Logs: what exactly |
| Percentiles | Need histograms; never average p99s |
| RED / USE / golden | Rate, errors, duration / utilisation, saturation, errors / latency, traffic, errors, saturation |
| Alerting | Page on symptoms and burn rate; causes go to tickets |
| Deploys | Rolling (cheap), blue-green (instant rollback, 2x), canary (small blast radius), shadow (no user risk) |
| Migrations | Expand, dual write, backfill, switch reads, contract; readers before writers |
| Capacity | Peak QPS / (tested QPS per instance × 60%), then add zone-loss headroom |
| Queueing | Latency ~ 1 / (1 - utilisation); 90% busy is 10x slower |
| Incidents | Detect, triage, mitigate, resolve, review; mitigate before diagnosing |
| Top incident cause | Change: code, config, flags, infra. Stage and mark all of it |
Next: putting all of this to work on real systems, in HLD Case Studies: Core.