Learnings

System design, part 7 of 12

Reliability and Operations

A design that works on a whiteboard is not yet a design that runs. This page covers what separates the two: how reliability is defined and measured, how a system is built to survive failures, how it is observed, shipped, sized and operated, and how to weave all of it into an interview answer. Interviewers at senior level listen for this explicitly.

Defining reliable

Part 1 of 7 · targets you can measure

"Highly available" means nothing until it has a number, a measurement and a consequence. Start every reliability discussion here.

SLI, SLO, SLA

Defining reliable

What it isExampleOwned by
SLI (indicator)A measured ratio: good events / valid eventsShare of checkout requests that return non-5xx in under 300 msEngineering
SLO (objective)Target for an SLI over a window99.9% of checkout requests good, over a rolling 28 daysEngineering plus product
SLA (agreement)Contract with a penalty if missed99.5% monthly uptime or 10% service creditBusiness and legal
The SLA is always looser than the SLO, so you get warned internally long before you owe anyone money.
  • Measure the SLI as close to the user as you can: load balancer logs beat server metrics, client telemetry beats both.
  • Never promise 100%. Every dependency, network and client you do not control is below 100%, and the last nine costs the most.

Choosing good SLIs

Defining reliable

Pick 2 to 4 SLIs per user journey, chosen by what the user would complain about, not by what is easy to scrape.

SLI typeGood eventFitsPitfall
AvailabilityRequest served without a server errorRequest/response APIsCounting 4xx as bad (client bugs) or ignoring requests that never reached you
LatencyServed faster than a threshold, at a percentile (p50, p99)Interactive pathsUsing the mean, which hides the tail; averaging percentiles across hosts
FreshnessData age below a bound (feed updated within 60 s)Pipelines, caches, replicas, search indexesMeasuring job success instead of data age
CorrectnessOutput matches a known answer (probes, reconciliation)Billing, ledgers, ML outputsOnly checked by users, months later
DurabilityWritten data still readableStorage systemsNever tested until a restore is needed

Senior signal: express latency SLOs as "99% of requests under 300 ms", not "average 100 ms". A p99 target on a page that fans out to 20 backends means each backend needs a much tighter tail.

Error budgets

Defining reliable

Error budget = 1 - SLO. A 99.9% SLO over 30 days allows 0.1% bad requests, about 43 minutes of full outage. It turns "reliability vs velocity" into a shared, numeric rule.

The policy is agreed in advance with product, so a freeze is a rule being followed, not an argument.
  • Unspent budget is a signal too: the SLO may be too loose, or the team too cautious. Spend it on risky migrations and chaos tests.
  • Budget consumed by a dependency still counts for the user; it is an input to the dependency's own SLO discussion.

Availability arithmetic

Defining reliable

AvailabilityDowntime / year/ 30-day month/ dayRealistic for
99% (two nines)3.65 days7.2 h14.4 minInternal batch tools
99.9%8.76 h43.2 min1.44 minMost product services in one region
99.95%4.38 h21.6 min43 sTypical managed cloud service SLA
99.99%52.6 min4.3 min8.6 sPayments, auth; needs multi-zone and automated failover
99.999%5.3 min26 s0.86 sMulti-region, no human in the recovery loop

Series: every hop must work

Availabilities multiply in series: the chain is worse than its weakest part.

Parallel: any one copy is enough

Two 99.5% copies give 1 - 0.005 squared = 99.9975%. The single database now dominates the total.
  • Series: A = A1 × A2 × ... A chain of ten 99.9% dependencies is about 99.0%: three nines decay to two.
  • Parallel: A = 1 - (1 - a)n, but only for independent failures with working failover. Shared config, a shared deploy or one zone breaks independence.

Interview tip: do this multiplication out loud once. It justifies why you add replicas, why you remove synchronous hops, and why a 99.99% target needs the database tier to be redundant too.

RTO and RPO

Defining reliable

RPO (recovery point objective): how much recent data you can afford to lose. RTO (recovery time objective): how long you can be down. Both are business inputs; they pick the replication and DR strategy.

RPO looks backwards from the disaster to the last safe copy; RTO looks forward to recovery.
  • RPO near zero needs synchronous replication (pay write latency) or a replicated log; RPO of minutes allows async replication or continuous log shipping.
  • RTO under minutes needs automated detection and failover; anything with "a human runs the runbook" is 15+ minutes in practice.

Designing for failure

Part 2 of 7 · assume every box dies

Every component fails; the design decides whether users notice. Remove single points of failure, decide how failover works, and decide what to switch off when things go wrong.

Redundancy levels

Designing for failure

Each ring survives a bigger failure and costs more: a dead host, a dead data centre, a dead region.
LevelSurvivesCostLatency impact
Instance (N+1, N+2)Host crash, bad deploy on one nodeLow: one or two spare instancesNone
Zone (multi-AZ)Power, network or cooling loss in one data centreModerate: cross-zone traffic is billed; capacity for N-1 zonesAbout 1 to 2 ms between zones; sync replication is affordable
RegionRegional outage, control-plane failure, disasterHigh: duplicate stack, egress, operational complexityTens to 100+ ms between regions; sync replication hurts

Failure mode: three zones, but peak needs all three of them busy. Losing one zone then overloads the other two and the outage spreads. Provision so N-1 zones can carry peak.

Single points of failure

Designing for failure

Walk the request path on your diagram and ask, for every box and arrow: "if exactly this dies, does the request fail?" Any yes without a mitigation is a SPOF.

The app tier is redundant, but the single LB, the replica-less database and the config service each take everything down.
  • Easy to miss: DNS, certificates (expiry), the secrets store, the auth service, the CI system you need to deploy a fix, a single NAT gateway, one on-call person.
  • Also a SPOF: a shared thing replicated everywhere, such as one global config push or one bad deploy to all regions at once.

Failover: active-passive and active-active

Designing for failure

Active-passive

One side serves; the other waits. Recovery time is detection plus promotion plus warm-up.

Active-active

Both sides serve. Losing one means the survivor must absorb all traffic, and concurrent writes must be reconciled.
Active-passiveActive-active
Failover timeSeconds to minutes (detect, promote, DNS TTL, cold caches)Near zero; route around the dead site
CostStandby capacity sits idle (can be smaller if warm)Each side needs headroom to carry 100% alone
DataOne writer; simpleMultiple writers need conflict handling or partitioned ownership
RiskStandby rots untested; split brain if both think they are primaryHidden cross-site dependencies; capacity mistakes cause cascades

Failure mode: split brain. Use fencing (a lease or epoch number the old primary must hold to write) so a partitioned old primary cannot keep accepting writes. Consensus and leases are covered in Distributed Systems Patterns.

Multi-region designs

Designing for failure

Go multi-region for latency to global users, for regional disaster survival, or for data residency. Each shape is decided by where writes go.

Read-local, write-global: reads are fast everywhere, writes pay one cross-region round trip, and EU readers may see slightly stale data.
ShapeWrites go toReplication choiceGainPay
Active-passive regionPrimary region onlyAsync to standby regionSimple; DRRPO = replication lag; RTO = failover drill
Read-local, write-globalOne home regionSingle leader, async read replicas per regionFast reads everywhereSlow writes for remote users; stale reads; use read-your-writes routing
Partitioned by user (geo-homing)The user's home regionEach partition has a leader in its home regionFast local writes; residencyCross-region users and global queries are hard; re-homing
Active-active multi-writerAny regionMulti-leader or leaderless, conflict resolution (LWW, CRDTs, app merge)Local writes, survive region lossConflicts, anomalies; only for data that merges well
Global consensusAny region, committed by quorumSynchronous quorum across 3+ regions (Spanner-style)Strong consistency, RPO 0Write latency of a cross-region quorum, 50 to 200 ms
Follow-the-sunRegion that owns the current time windowOwnership moves on a schedule; async catch-upWrites local to the active workforceHandover windows; mostly for internal tools and ops

Senior signal: split the data. Keep money and inventory single-writer or consensus-backed, and let carts, likes and presence be active-active with merges. One replication model for the whole system is rarely right. Replication and conflict mechanics are in Data Systems.

Disaster recovery strategies

Designing for failure

StrategyWhat runs in the DR regionRTO (approx.)RPO (approx.)Cost
Backup and restoreNothing; backups copied thereHours to a dayHours (last backup); minutes with continuous log backup$
Pilot lightData replicated; compute off or minimalTens of minutes to hoursSeconds to minutes$$
Warm standbyFull stack, scaled down, serving no usersMinutesSeconds$$$
Multi-site active-activeFull stack, serving usersNear zeroNear zero to seconds$$$$
  • Pick per data set and per service, from the business RTO / RPO, not one strategy for the whole company.
  • Infrastructure as code is what makes pilot light and warm standby realistic: the DR stack is the same templates with different sizes.

Backups that are actually tested

Designing for failure

A backup you have never restored is a hope. Replication is not a backup: it copies a bad DELETE to every replica in milliseconds.

  • 3-2-1: three copies, two media or services, one off-site (another region or account).
  • Immutable / separate account: object lock or a separate account so a compromised credential or ransomware cannot delete them.
  • Point-in-time recovery: snapshot plus write-ahead log lets you restore to just before the bad change.
  • Restore drills on a schedule: automated restore into a scratch environment, row-count and checksum checks, and the measured duration recorded as your real RTO.

Graceful degradation and load shedding

Designing for failure

When capacity or a dependency is lost, serve a smaller product instead of no product, and reject excess work early and cheaply instead of slowing down for everyone.

Shed at the edge by priority, and fall back per optional feature, so the core action keeps working.
  • Degradation examples: show cached prices with a "may be out of date" note, hide recommendations, disable search filters, switch to read-only mode.
  • Shedding: bound queues and concurrency; drop work whose client has already timed out; prefer checkout traffic over crawlers and batch jobs.
  • Pre-build the switches (feature flags, kill switches) and test them; you will not write them safely during an incident.

Critical vs optional dependencies

Designing for failure

Classify every dependency of every request. The answer decides timeouts, fallbacks and whether its outage is your outage.

Only solid edges can fail the request. Dashed edges get short timeouts and a fallback, or move off the request path entirely.
ClassIf it failsTreatment
CriticalThe request failsRedundant, SLO at least as tight as yours, retries with budget, circuit breaker that fails fast
Optional (soft)Feature hidden or defaultedTight timeout, fallback value, never retried at length
AsynchronousWork delayedQueue with retries and a dead-letter queue

Circuit breakers, retries with backoff and jitter, timeouts, bulkheads and backpressure are covered in Distributed Systems Patterns.

Observability

Part 3 of 7 · knowing what it is doing

Monitoring tells you that something is wrong; observability lets you ask why, including questions you did not plan for. Both are built from three signals.

Metrics, logs, traces

Observability

SignalAnswersShapeCost driverWeak at
MetricsIs something wrong, and how much? Trends, alertsNumeric time series with a few labelsLabel cardinality (never label by user id)Explaining one specific request
LogsWhat exactly happened in this component?Timestamped events, ideally structuredVolume: bytes ingested and retainedCross-service flow; aggregation is expensive
TracesWhere did the time go across services for this request?Tree of timed spans per requestSpan volume; usually sampledRare events if sampled away

The workflow: an alert fires on a metric, an exemplar or filter leads to a slow trace, the trace id leads to the logs of the failing span. Link them with shared ids.

Metric types

Observability

TypeBehaviourExampleQuery as
CounterOnly goes up (resets on restart)http_requests_total{status="500"}Rate per second
GaugeGoes up and down; a current valueQueue depth, memory in use, connectionsCurrent value, max, min
HistogramCounts observations into bucketsRequest duration buckets 10, 50, 100, 300, 1000 msPercentiles estimated from bucket rates
  • You cannot average percentiles: the mean of 20 hosts' p99 is not the fleet p99. Histograms can be summed across hosts, then the percentile computed, which is why latency SLOs need them.
  • Choose bucket edges around your SLO threshold (for example one at exactly 300 ms) so "share under 300 ms" is exact.

RED, USE and the four golden signals

Observability

MethodForMeasure
REDRequest-driven servicesRate, Errors, Duration
USEResources: CPU, memory, disk, network, pools, queuesUtilisation, Saturation (queued work), Errors
Four golden signals (Google SRE book)Any user-facing systemLatency, Traffic, Errors, Saturation

RED tells you users are hurting; USE tells you which resource is why. Separate the latency of successful and failed requests: fast errors make a broken service look quick.

Distributed tracing

Observability

A trace is one request's journey; a span is one timed operation in it with a parent span. Context propagation passes trace id and parent span id across process boundaries, usually in the W3C traceparent header or in message headers for queues.

Every hop forwards the same trace id. The trace view shows Payments spent 180 of the 195 ms.
  • Sampling: head sampling (decide at the start, for example 1%) is cheap but misses rare errors; tail sampling (decide after the trace ends) keeps all slow and failed traces at the cost of buffering.
  • Propagation breaks silently at thread pools, queues and third-party SDKs; check that async consumers continue the trace.
  • OpenTelemetry is the common vendor-neutral standard for emitting all three signals.

Structured logs and correlation ids

Observability

Log events as key-value records, not sentences, so they can be filtered and aggregated. Every line carries the request's correlation id (often the trace id).

  • Generate the id at the edge, return it to the client (support tickets can quote it), propagate it everywhere including queue messages.
  • Levels with meaning: error = someone should look; warn = degraded but handled; info = business events; debug off in production or sampled.
  • No secrets or raw personal data in logs; hash or omit. Logs are often kept longer and seen by more people than the database.

Python specifics (the logging module, structured output, debugging) are in Debugging and Logging.

Alerting

Observability

DoAvoid
Page on symptoms users feel: SLO burn, error rate, latencyPaging on causes: CPU 80%, one pod restarted, disk 70%
Every page is urgent, actionable and has a runbook linkAlerts nobody acts on (they train people to ignore pages)
Causes go to tickets or dashboardsDuplicate alerts for one incident from every layer
Review alert volume weekly; delete or fix noisy onesStatic thresholds tuned once and forgotten

Burn-rate alerts: burn rate is how fast you are spending the error budget relative to plan (1 = exactly used up at the end of the window). A common setup pages when the 1-hour burn rate exceeds about 14 (2% of a 30-day budget gone in an hour, confirmed over a short window too) and opens a ticket at a slower burn over 6 hours or more.

Failure mode: alert fatigue. A team paged ten times a night stops reading pages, and the real one is missed. Fewer, symptom-based pages are a reliability feature.

Dashboards worth having

Observability

  • Service overview: RED per endpoint, SLO and remaining error budget, deploy markers on every graph.
  • Dependencies: latency and error rate of each downstream call, circuit breaker state.
  • Resources: USE for CPU, memory, connection pools, thread pools, queue depth and consumer lag.
  • Business: orders per minute, sign-ups, payments succeeded. Often the first to show a silent failure.

A dashboard should answer one question in ten seconds at 3 a.m. Dozens of unlabelled graphs are not observability.

Shipping safely

Part 4 of 7 · change without outages

Most outages start with a change. Safe delivery limits how many users a bad change reaches and how quickly it can be undone.

Deployment strategies

Shipping safely

Rolling

Instances are replaced a batch at a time; v1 and v2 serve side by side during the rollout.

Blue-green

A full second environment; switching the router is the deploy, switching back is the rollback. The database is shared, so the schema must suit both.

Canary

A small slice of real traffic judges the release automatically before everyone gets it.

Shadow (dark launch)

v2 sees real traffic but its responses are discarded. Writes and side effects such as emails or charges must be disabled in the shadow.
StrategyUser riskExtra costRollback speedWatch out for
RecreateDowntime for allNoneSlow: redeploy old versionOnly for dev or tolerated maintenance windows
RollingGrows with each batchLittle (surge capacity)Medium: roll back batch by batchMixed versions; must be compatible
Blue-greenAll at once on switchUp to 2x during releaseSeconds: flip the routerShared DB and caches; sessions on switch
CanarySmall slice firstSmallFast: drain the canaryNeeds good metrics and enough canary traffic to be significant
ShadowNone (responses discarded)Up to 2x for mirrored pathsNothing to roll backSide effects; only validates, never releases

Feature flags and progressive delivery

Shipping safely

Flags separate deploy (code is on servers) from release (users see it). Progressive delivery widens the audience in steps, gated by metrics.

Turning a flag off is a rollback that takes seconds and needs no deploy.
  • Kinds: release flags (short-lived), ops kill switches (permanent), experiment flags (A/B), permission flags (plans, tenants).
  • Trade-off: every flag doubles a code path. Give release flags an owner and an expiry date, and delete them.
  • Flag evaluation must not be a critical dependency: cache flags locally, with safe defaults if the flag service is down.

Zero-downtime schema migrations

Shipping safely

Old and new code run together during any rolling deploy, so every schema state must work with both. Example: rename name to full_name.

Each step is its own deploy and each is reversible until the final drop.
  • Backfill in small batches with throttling; one giant UPDATE locks rows and floods replication.
  • Watch for locking DDL: adding a column with a default, adding an index or a NOT NULL constraint can lock a big table on some engines and versions; use online schema change tools or concurrent index builds.
  • Contract only after every service, job and old mobile client has stopped reading the old column.

Backward-compatible changes

Shipping safely

SafeBreaking
Add an optional field or endpointRemove or rename a field
Accept a new enum value on read (ignore unknowns)Send a new enum value to old readers that reject it
Widen what you acceptNarrow validation, change a type or units
New message version alongside the oldChange the meaning of an existing field

Rule: deploy readers before writers. Consumers must tolerate the new shape before any producer emits it. For events, a schema registry with compatibility checks enforces this. API versioning is in API and Communication Design.

Rollback vs roll-forward

Shipping safely

  • Roll back first when the last change is the likely cause. Mitigate, then investigate. Rollback must be one command or one flag flip and rehearsed.
  • Roll forward when rollback is unsafe: a data migration already ran, a new format was written that old code cannot read, or the fix is tiny and well understood.
  • Design so rollback is always possible: expand-contract migrations, readers before writers, flags around risky paths.

Senior signal: "every deploy must be reversible for at least one release" is a design constraint, not an ops detail. It shapes migrations and message formats.

CI/CD pipeline

Shipping safely

Build once, promote the same artifact, and widen the blast radius in waves with automatic stops.
  • Deploy regions or cells one wave at a time with bake time between, so a bad release never reaches everyone at once.
  • Infrastructure as code (Terraform, CloudFormation and similar): servers, networks and config are versioned, reviewed and reproducible, which is what makes DR stacks and identical environments possible.

Capacity and performance

Part 5 of 7 · enough, but not too much

Turn traffic estimates into machines, keep headroom for peaks and failures, and know where the system will break first.

Capacity planning

Capacity and performance

Measure per-instance capacity, do not guess it, then add headroom for spikes and for losing a zone.

Worked example: 10 M daily users, 20 requests each, peak 3x average; one instance handles 800 QPS within the latency SLO in a load test and runs at 60% of that; 3 zones, must survive losing one.

  • 15 instances carry peak; 24 are deployed so any two zones can carry it. Zone redundancy costs 60% more here, and that is the price of the SLO.
  • Repeat for each tier: the database, cache memory and queue throughput usually run out before stateless compute.
  • Plan for growth: forecast 6 to 12 months ahead for anything with a long lead time (database shards, reserved capacity).

Autoscaling

Capacity and performance

A control loop with a lag: new instances take time to boot and warm up, so scale out early and scale in slowly.
KindHowGood forWeak at
Reactive (target tracking)Keep a metric near a target, for example 60% CPU or 400 RPS per instanceGradual changesSudden spikes faster than boot time
Scheduled / predictiveScale ahead of known patterns (daily peak, a sale, a match)Predictable loadSurprises
Queue-basedWorkers sized by backlog / acceptable drain timeAsync consumersPartitioned queues cap useful consumers
  • Cooldowns and asymmetric thresholds stop flapping. Keep a minimum that survives a zone loss.
  • The stateful tier is the hard part: adding a database node means moving data (resharding, rebalancing) which takes hours and adds load exactly when you are busiest. Plan stateful capacity ahead; autoscale the stateless tiers.
  • Scaling out the app can overload the database (more connections); use pooling and cap the fleet.

Load testing types

Capacity and performance

TypeShape of loadFinds
LoadExpected peak, heldWhether you meet the SLO at planned capacity
StressIncrease until it breaksThe breaking point and how it fails (gracefully or by collapse)
SoakNormal load for hours or daysLeaks: memory, connections, file handles, growing queues
SpikeSudden jump, for example 10x in a minuteAutoscaling lag, cold caches, thundering herds

Use production-like data sizes and realistic request mixes; a test against an empty database or a single hot key measures the wrong thing.

Finding bottlenecks

Capacity and performance

  • Start from the slowest span in traces, then apply USE to that component: which resource is saturated?
  • Usual suspects, in rough order: database (slow queries, missing indexes, lock contention, connection pool), hot keys or partitions, synchronous calls to slow dependencies, thread or connection pool limits, GC pauses, network egress.
  • Profile under load (CPU flame graphs, allocation profiles); the code that is slow at 1 QPS is often not the one slow at 5,000.
  • Fix the biggest one, measure again. The bottleneck moves; optimising a non-bottleneck changes nothing.

Queueing intuition

Capacity and performance

In a simple queue model (M/M/1), mean response time is service time / (1 - utilisation). Latency stays flat until about 70% and then climbs steeply.

Utilisation50%70%80%90%95%99%
Response time vs idle2x3.3x5x10x20x100x
  • At 100% utilisation the queue grows without bound: every request times out, clients retry, and load rises further. That is why it is a failure, not efficiency.
  • This is why targets sit at 50 to 70%, why queues need bounds, and why load shedding beats queueing during overload.
  • Little's law: items in system = arrival rate × time in system. At 2,000 QPS and 50 ms, about 100 requests are in flight; size pools from that.

Cost as a design constraint

Capacity and performance

CostGrows withLevers
ComputePeak load, headroom, idle standbyRight-size instances, autoscale, reserved or committed capacity for the base, spot for batch
StorageData volume times replicas times retentionTiered storage (hot SSD, warm, cold object store, archive), TTLs, compression
EgressBytes leaving a region or the cloud; cross-zone trafficCDN, compression, keep chatty services in one zone, avoid cross-region fan-out
ObservabilityLog volume, metric cardinality, trace volumeSampling, retention tiers, drop debug logs

Senior signal: mention cost when choosing a reliability level. "Active-active in three regions roughly triples the infrastructure bill; for this product a warm standby meets the RTO" is the kind of trade-off interviewers want to hear.

Operating it

Part 6 of 7 · people and process

Reliability is also a practice: how incidents are handled, learned from and prevented.

Incident response

Operating it

Mitigation (roll back, fail over, shed load) comes before diagnosis. Users care that it stopped, not why yet.
  • Roles: incident commander (coordinates, decides), operations lead (hands on keyboard), communications lead (status page, stakeholders). Small incidents may combine them.
  • Severity levels agreed in advance (for example SEV1: major user impact, all hands) decide who is paged and how often to update.
  • Keep a timeline as you go in the incident channel; it becomes the postmortem.
  • Key measures: time to detect, time to mitigate. MTTR matters more than MTBF for most systems.

On-call and runbooks

Operating it

  • Rotation of at least 5 to 8 people per schedule, a primary and a secondary, a clear escalation path, and a handover note each shift.
  • Measure load: pages per shift and night pages. A sustained couple of incidents per shift means the system needs engineering, not more heroes.
  • Runbooks: linked from each alert; state what the alert means, how to confirm user impact, safe mitigations (commands, dashboards, flags), and when to escalate. Automate steps that are run often.
  • The team that builds a service is on call for it, or at least owns its alerts; it closes the feedback loop.

Blameless postmortems

Operating it

Blameless means asking why the system let a reasonable person make the mistake, not who made it. People hide information when they fear blame, and the real causes stay hidden.

SectionContains
Summary and impactWho was affected, how much, how long; error budget spent
TimelineDetection, key decisions, mitigation, resolution, with timestamps
Contributing factorsUsually several: the trigger, the latent bug, the missing guard, why detection was slow
What went well, what was luckyLuck is a risk to remove
Action itemsSpecific, owned, dated, prioritised; tracked to done

Failure mode: action items that say "be more careful". Good actions change the system: a guard rail, an automated check, an alert, a safer default.

Chaos engineering and game days

Operating it

A chaos experiment is a test of a stated hypothesis, with an abort switch, not random breakage.
  • Faults to inject: kill instances, add latency to a dependency, drop a zone, fill a disk, expire a certificate, exhaust a connection pool, fail DNS.
  • Game days: scheduled exercises where a team runs a failure scenario (often a DR failover) end to end, including the humans and runbooks.
  • Start in staging, then production during working hours with small scope and error budget to spare.

Change management

Operating it

Changes (code deploys, config pushes, feature flags, infrastructure, certificate and DNS updates) are widely reported as the largest single trigger of production incidents. Treat every change, not only code, as a deploy.

  • Config and flags go through review, validation, staged rollout and quick rollback, just like code. A global config push is the classic way to take down every region at once.
  • Every change is recorded with who, what and when, and shown as a marker on dashboards, so "what changed?" is answered in seconds.
  • Freeze or slow changes during peak events and when the error budget is spent.

Production readiness checklist

Operating it

AreaReady means
SLOsSLIs defined and measured; SLO and error budget policy agreed
RedundancyNo SPOF on the critical path; N-1 zone capacity; failover tested
DependenciesEach classified; timeouts, retries with budget, circuit breakers, fallbacks
ObservabilityRED dashboards, structured logs with correlation ids, tracing, deploy markers
AlertingSymptom-based pages with runbooks; owner and on-call rotation
DeliveryAutomated pipeline, canary or staged rollout, one-step rollback
DataBackups with a tested restore; RPO and RTO met in a drill; migrations expand-contract
CapacityLoad tested at expected peak plus headroom; autoscaling limits; quotas raised
OverloadRate limits, load shedding, bounded queues
SecuritySecrets managed, least privilege, certificate expiry monitored

In the interview

Part 7 of 7 · saying it out loud

Reliability is not a separate section at the end of an answer; it threads through it. These chapters show how, and summarise the page.

Weaving reliability into an HLD answer

In the interview

Phase of the answerWhat to say
RequirementsName SLOs up front: "99.9% for reads, 99.99% for payments, p99 under 200 ms; RPO zero for orders, minutes for analytics."
High-level diagramMark the single points of failure yourself before the interviewer does, and say how each is removed.
Deep diveFor each component: what happens when it dies, how it is detected, what the user sees.
ScaleCapacity arithmetic with headroom for a zone loss; where the stateful bottleneck is.
Wrap-upHow you would deploy it (canary, flags, expand-contract), observe it (SLO dashboards, burn-rate alerts) and recover it (DR strategy that matches the RTO).

Senior signal: mid-level answers say "add replicas for availability". Senior answers say which failure each replica covers, what failover costs in time and data, and what the user sees meanwhile.

Component fails, then what?

In the interview

A sample e-commerce architecture, then the table you should be able to produce for it.

Every box here appears in the table below; the payment provider is a critical dependency you do not control.
Component failsUser impactMitigation
One API instanceNone; a few requests retriedHealth checks remove it; N+2 capacity
A whole zoneBrief errors, then noneTwo zones sized to carry peak; LB stops routing to the dead zone
Cache clusterSlower pages; risk of DB overloadReplicated cache; request coalescing; shed low-priority reads; warm on recovery
Primary databaseWrites fail for seconds to a minuteAutomated failover to synchronous standby (RPO 0); fencing of the old primary
QueueEmails and fulfilment delayedReplicated queue; outbox table holds events until it returns
WorkersBacklog grows; no immediate user impactAutoscale on lag; idempotent consumers; DLQ for poison messages
Payment providerCannot payTimeout and circuit breaker; second provider or "order accepted, payment pending" with retry
CDNSlow or failing static assetsOrigin can serve directly at reduced scale; DNS switch to a second CDN
Whole regionOutage until failoverWarm standby region; RTO minutes, RPO = async lag; rehearsed by game day
Bad deployErrors for the canary sliceAutomatic canary abort and rollback

Cheat sheet

In the interview

TopicRemember
SLI / SLO / SLAMeasured ratio / internal target / contract with penalty; SLA looser than SLO
Error budget1 - SLO; 99.9% is about 43 min per 30 days; exhausted means freeze features
Nines99.9% = 8.8 h/yr; 99.99% = 53 min/yr; 99.999% = 5 min/yr
CompositionSeries multiplies (ten 99.9% hops is about 99.0%); parallel 1 - (1 - a)^n if independent
RPO / RTOData you can lose / time you can be down; pick replication and DR from them
RedundancyInstance, zone, region; N-1 zones must carry peak
FailoverActive-passive: cheap, slower, split-brain risk. Active-active: fast, 2x headroom, conflicts
DR ladderBackup-restore (hours) to pilot light, warm standby (minutes), multi-site (near zero)
BackupsReplication is not a backup; 3-2-1; restore drills give the real RTO
OverloadDegrade optional features; shed low priority early; bound queues
SignalsMetrics: is it broken. Traces: where. Logs: what exactly
PercentilesNeed histograms; never average p99s
RED / USE / goldenRate, errors, duration / utilisation, saturation, errors / latency, traffic, errors, saturation
AlertingPage on symptoms and burn rate; causes go to tickets
DeploysRolling (cheap), blue-green (instant rollback, 2x), canary (small blast radius), shadow (no user risk)
MigrationsExpand, dual write, backfill, switch reads, contract; readers before writers
CapacityPeak QPS / (tested QPS per instance × 60%), then add zone-loss headroom
QueueingLatency ~ 1 / (1 - utilisation); 90% busy is 10x slower
IncidentsDetect, triage, mitigate, resolve, review; mitigate before diagnosing
Top incident causeChange: code, config, flags, infra. Stage and mark all of it

Next: putting all of this to work on real systems, in HLD Case Studies: Core.