SRE practices: SLOs, error budgets, incidents

Why this matters

Staff engineers are responsible for systems' lifetime behaviour, not just their initial implementation. SRE practices are the discipline of running software reliably: defining what "reliable" means quantitatively (SLOs), trading reliability against velocity (error budgets), responding to failures (incident response), and learning from them (post-mortems).

The shift here is from "code that works" to "code that's known to work, in production, at 3am, when nothing seems quite right." That's a different skill.

SLIs, SLOs, SLAs — disambiguation

• SLI (Service Level Indicator): a measurement. "Fraction of requests completing in < 500ms over the last 5 minutes."
• SLO (Service Level Objective): an internal target for the SLI. "99.9% over 30 days."
• SLA (Service Level Agreement): a customer-facing contract with consequences for breach. Typically much weaker than SLO. "99.5% — we refund a month of fees if breached."

Internal SLOs are stricter than external SLAs. The SLO is what engineering targets; the SLA is what marketing/legal commits to.

Choosing the right SLI

Most teams over-instrument and pick the wrong SLIs.

Bad SLI: "CPU utilisation under 80%." This is a resource metric, not a user-experience metric. Users don't care about CPU; they care if their request succeeded.

Good SLI: "Fraction of requests to /checkout that complete in under 1 second with a 2xx status code." This directly measures user experience.

Pick SLIs that are user-facing. Categories that work:

• Availability: fraction of requests succeeding (2xx/3xx response). Define carefully — is 4xx a "success" (user error) or "failure" (your bug)?
• Latency: fraction of requests completing within a threshold.
• Correctness: fraction of requests returning correct results. Hardest to measure; often via shadow testing.
• Freshness: fraction of reads returning data updated within X seconds (for async pipelines).
• Throughput: requests served per second (for systems with capacity constraints).

How to pick the SLO threshold

Three approaches, all valid:

1. Empirical: measure current performance, set SLO slightly below it. "We've been doing 99.5%; let's commit to 99.5%."
2. User-driven: ask "what fraction of failures would users notice?" Set SLO to match.
3. Business-driven: what reliability does the product require to function? A payment service might need 99.99%; an analytics dashboard might be fine at 99%.

Don't over-promise. 99.99% (52 minutes of downtime per year) is a different operational commitment than 99.9% (8.76 hours per year). Each "nine" you add roughly doubles operational cost. Most internal services should be 99.5% or 99.9%. Critical revenue paths might be 99.99%. 99.999% is essentially impossible for stateful, multi-component systems without massive investment.

Error budgets — the social contract

Error budget = 100% − SLO.

If your SLO is 99.9%, your error budget is 0.1% of requests. If you serve 1M requests/day:

• Per day: 1,000 requests can fail
• Per week: 7,000
• Per month (30 days): 30,000

The error budget is permission to take risks. If you've used 10% of your budget for the month, you can be aggressive with deploys and experiments. If you've used 90%, you should freeze risky changes and focus on reliability.

This is the SRE "social contract": product can push for features; SRE can push back when error budget is depleted. Without an error budget, this becomes a subjective argument; with one, it's a number.

Burn-rate alerting (modern best practice)

Naive alerting: alert when error rate exceeds some threshold. Two problems:

1. Slow burns get missed: 1% error rate over a week silently depletes your monthly budget; never triggers a threshold alert.
2. Fast burns alert too late: by the time you notice, the budget is spent.

Multi-window, multi-burn-rate alerting (from the Google SRE Workbook):

The shorter windows catch fast-burning catastrophes; the longer windows catch slow-burning chronic issues. Combine them and you alert at the right urgency.

Incident response — the operational pattern

An incident is any unplanned event that degrades or risks degrading the service. The discipline of responding is structured.

Roles

• Incident commander (IC): drives the response. Doesn't necessarily fix; coordinates fixers. One person; this role is sacred.
• Operations lead: actually executes mitigations. May be the IC if small incident.
• Communications lead: updates stakeholders (customer success, leadership, status page). Frees engineers from the comms burden.
• Scribe: maintains a running timeline. Critical for post-mortem.

The response phases

1. Detect: alert fires, or someone reports. Acknowledge.
2. Assess severity: how many users affected? Revenue impact? Is it growing?
3. Stabilise: stop the bleeding. Roll back the deploy, increase capacity, failover, etc. Mitigation BEFORE root-cause analysis. Get back to a safe state.
4. Investigate: now figure out root cause. With the bleeding stopped, you have time to think.
5. Fix: apply the durable fix.
6. Communicate: keep stakeholders informed throughout. Status page, internal channels, post-incident.
7. Post-mortem: within 48 hours, write up what happened and what changes.

The most important lesson: mitigate before you root-cause. The instinct is to fix the bug; the discipline is to first stop the damage. Roll back, scale up, failover — anything to restore service — even if you don't yet know why.

Blameless post-mortems

The standard format:

1. Summary (one paragraph): what happened, impact, duration.
2. Timeline: minute-by-minute, with timestamps and quoted messages. Don't sanitise.
3. Impact: how many users, requests, revenue. Quantify.
4. Root cause: 5 whys analysis. Or contributing factors if not a single root cause.
5. What went well: detection speed, response, communication. Praise specific behaviour.
6. What didn't go well: where the response failed or was slow.
7. Action items: who, what, when. Tracked to completion.

Blameless means: focus on systems, not individuals. "Engineer X deployed broken code" is wrong framing; "the deploy pipeline allowed broken code through" is right. Blameless post-mortems aren't soft — they're harder. They demand systemic fixes, not human ones.

Toil reduction

Toil = manual, repetitive operational work that doesn't grow value over time. Examples: manually restarting a flaky service, manually rotating credentials, manually onboarding new users.

Google's SRE book sets a target: SRE teams should spend no more than 50% of their time on toil. The other 50% should be on engineering work that reduces toil.

How to identify and reduce:

1. Log every operational task and its time cost.
2. If a task repeats more than ~3 times, automate it.
3. Build self-service: instead of "ask SRE to restart the service," let app teams restart their own.
4. Build observability: most toil comes from "we don't know what's happening." Better dashboards reduce toil.

Runbooks

For every alert, there should be a runbook: "if this alert fires, do these steps." Linked from the alert message itself.

Good runbook structure:

1. What the alert means (in plain English)
2. Likely causes
3. Diagnostic steps (dashboards to check, commands to run)
4. Mitigation steps (specific commands, with safety notes)
5. Escalation: who to involve if mitigation fails

Without runbooks, every page becomes a "figure it out under stress" exercise. With runbooks, on-call shifts are tolerable and knowledge spreads across the team.

Applied scenarios

Further reading