Observability

Why this matters

You can't operate what you can't see. Junior engineers write code; senior engineers write code that they can debug at 3am after a year of memory drift, partial deploys, traffic spikes, and forgotten edge cases. The difference is observability — the discipline of designing systems so their internal state is recoverable from the outside.

Most production issues don't look like "the code is wrong." They look like "everything seems fine, but p99 latency is up 4× and we can't tell which downstream is causing it." Solving these problems requires observability that's been designed in, not bolted on. Adding logs and metrics after an outage is the SRE equivalent of "the burglar broke in and now I'm thinking about getting a lock."

Observability is the topic where Staff engineers most clearly separate from SDE2s. Staff engineers design for unknown-unknowns: things that haven't gone wrong yet but might. SDE2s tend to instrument what they think will break, miss what actually breaks, then have to retrofit instrumentation under pressure.

The three pillars (and their critique)

The classical model says observability stands on three pillars: metrics, logs, and traces. Each has its strengths and weaknesses.

Metrics

Aggregated, numeric, low-cardinality time-series data. Things like "requests per second," "error rate," "p99 latency," "CPU usage." Stored cheaply because they're aggregated; queried fast for dashboards.

Pros: cheap (typically pennies per million data points), fast queries (sub-second dashboard rendering), perfect for alerting.

Cons: aggregation destroys detail. If your p99 latency spiked, metrics tell you that it spiked but not which requests or why. You can't drill from a metric down to an example. Also: cardinality limits. Adding "user_id" as a metric label explodes cardinality (millions of unique values), which kills Prometheus performance and explodes storage cost.

Tools: Prometheus (pull-based, ubiquitous), Datadog (paid SaaS), CloudWatch (AWS-native), StatsD (push-based older standard), VictoriaMetrics (Prometheus-compatible at scale), Mimir.

Logs

Per-event text or structured records. "Order 123 placed by user 456 at 2026-05-16T14:32:11Z." High detail; expensive to store and search.

Pros: high fidelity. Captures the specific event with all its context. Great for forensics.

Cons: expensive at scale. Searching billions of log lines is slow and costly. Untyped — strings vs structured. Disconnected from each other unless you propagate request IDs.

Modern best practice: structured logging (JSON or similar), with request IDs / trace IDs in every log line so you can correlate across services.

Tools: ELK stack (Elasticsearch + Logstash + Kibana), Loki (Grafana's "Prometheus for logs"), Datadog Logs, Splunk, CloudWatch Logs.

Traces

End-to-end request paths across services. A single user click might touch 30 services; a trace shows the parent-child relationships, timing of each, and where time is spent.

Pros: irreplaceable for understanding distributed system behaviour. You'll never debug a 30-service microservices outage with metrics alone.

Cons: cost. Storing every span of every request at high traffic is expensive, so you sample. But sampling means you might not have the trace you need when you need it.

Standard: OpenTelemetry. Vendor-neutral SDK + protocol. Send your traces to Jaeger, Tempo, Datadog APM, Honeycomb, Lightstep — pick your backend.

Two sampling strategies:

• Head-based: decide at the start of the request whether to trace it. Simple; predictable cost; bad because you decide before knowing whether the request is interesting (errored, slow, etc.).
• Tail-based: collect every span, decide whether to keep the trace after it's complete. Better signal (keep all errors, slow requests); more complex (need a buffer per trace ID until the trace is "complete").

The modern critique: pillars are insufficient

Charity Majors and Honeycomb have argued for a different model: instead of three siloed data types, emit wide structured events — one event per significant unit of work, with all the relevant context (user_id, request_id, latency, error_code, downstream_call_count, etc.) attached. High cardinality on every dimension.

The argument: most production debugging is "find me requests where X is true and Y is unusual." Metrics can't answer this (aggregation destroys it). Logs can sort of answer it but require text search. Traces give you the request flow but cardinality is limited.

One wide event per request — say, with 50 dimensions, all high-cardinality — gives you a queryable database of production behaviour. You can ask: "show me requests where user_tier='premium' AND endpoint='/checkout' AND p99_downstream_call=='payment-svc' AND error_code='502'." Metrics can't do this. Honeycomb, Lightstep, Datadog's BPF tools, eBPF generally are heading this direction.

USE method — for resource analysis

Brendan Gregg's framework. When investigating a slow or broken system, for every resource (CPU, memory, disk, network, etc.):

• Utilisation: what fraction of the resource is in use?
• Saturation: how much extra work is queued waiting for the resource?
• Errors: are there error events related to the resource?

Example: investigating slow database queries.

• CPU utilisation: top, mpstat. If pinned at 100%, you've found a culprit.
• CPU saturation: run-queue length (vmstat r column). High = CPU bound.
• Memory utilisation: free, vmstat. Near limit = swapping risk.
• Memory saturation: swap activity (vmstat si/so columns). Active swap = death.
• Disk utilisation: iostat %util. High = I/O bound.
• Disk saturation: iostat avgqu-sz. High = queueing.
• Network utilisation: sar -n DEV. Approaching NIC limit = bottleneck.
• Errors: dmesg, /var/log/messages, network errors, disk errors.

USE is exhaustive — you check every resource. Most engineers wing it; USE makes it systematic. Read Brendan Gregg's site for the full method including kernel-level resources.

RED method — for services

Tom Wilkie's framework for request-driven services:

• Rate: requests per second.
• Errors: rate of errored requests.
• Duration: distribution of latencies (median, p95, p99).

These three are sufficient to know if a service is healthy. Every service should expose them; every dashboard should display them as the top row.

The RED method matches what users care about: "Am I getting served? Is my request succeeding? Is it fast?" — translated to operational metrics.

Cardinality — the cost dimension

Cardinality is the number of unique values for a label. Metrics systems struggle with high cardinality.

Example: a counter http_requests_total with labels {method, status, endpoint}:

• method: ~5 values (GET, POST, PUT, DELETE, PATCH)
• status: ~50 values
• endpoint: ~100 values
• Total cardinality: 5 × 50 × 100 = 25,000 time series

That's fine. But add user_id as a label, with 1M users: 25,000 × 1M = 25 billion time series. Prometheus melts.

Rule: keep cardinality of metrics labels in the thousands, not millions. For high-cardinality dimensions (user_id, request_id), use logs or wide events instead.

SLOs and error budgets — running services with intent

SLO = Service Level Objective. A measurable promise about service behaviour. "99.9% of requests will complete in under 500ms over a 30-day window."

Three components:

• SLI (Indicator): a measurable property. "Fraction of requests completing in under 500ms."
• SLO (Objective): a target for the SLI. "99.9% over a rolling 30-day window."
• Error budget: 100% minus SLO. "0.1% of requests can be slow/failed before we breach SLO."

Error budgets in practice

If your SLO is 99.9%, your error budget is 0.1% of requests per 30 days. If you serve 1M requests/day, that's 30M/month, and 0.1% = 30,000 "failed" requests allowed per month.

The error budget is the permission to take risks. If you've used 80% of your budget already, you should slow down on risky deploys. If you've used 0%, you can be aggressive. This is how SRE teams negotiate with product teams: "we're below SLO, all engineering effort goes to reliability."

Burn rate alerting

Naive alerting: "alert if error rate exceeds X." Misses slow burn (small but persistent error rate that depletes budget over weeks).

Better: multi-window, multi-burn-rate alerting. Alert if:

• You're burning > 14.4× normal rate over a 1h window (would deplete budget in 2 days)
• You're burning > 6× normal rate over a 6h window (deplete in 5 days)
• You're burning > 3× normal rate over a 24h window
• You're burning > 1× normal rate over a 72h window

Combinations catch fast-burning catastrophic failures and slow-burning chronic ones. Read the Google SRE Workbook chapter on alerting for the math.

What good observability design actually looks like

For every new service, before writing code:

1. What are this service's RED metrics? Where are they emitted?
2. What downstream calls does it make? Each one needs latency tracking and error attribution.
3. What's the trace context propagation strategy? OpenTelemetry SDK + middleware.
4. What structured fields go in every log line? At minimum: request_id, trace_id, user_id (if applicable), tenant_id (if applicable), service version.
5. What's the SLO? Where does it live? Who watches it?
6. What dashboards do I need on day one? RED, downstream calls, resource USE.
7. What alerts? Multi-window burn-rate on SLO. Resource saturation. Downstream-specific errors above threshold.

Then write the code such that emitting all this is automatic (middleware, decorators, instrumentation libraries) rather than per-endpoint manual work. The cost of "instrumented by default" is small at design time; retrofitting instrumentation post-launch is 5-10× the effort.

Distributed tracing in depth

A trace is a tree of spans. Each span represents a unit of work (an HTTP call, a DB query, a function execution). Spans have parent-child relationships.

Trace: user clicks "place order"
├── Span: API gateway (50ms)
│   ├── Span: auth service (10ms)
│   └── Span: order service (40ms)
│       ├── Span: inventory check (5ms)
│       ├── Span: payment service (20ms)
│       │   └── Span: stripe call (15ms)
│       └── Span: inventory decrement (3ms)

Context propagation is how spans link across services. Each request carries a trace_id and the calling span's span_id as parent. Standard headers (W3C Trace Context): traceparent, tracestate.

Common pitfalls:

• Async work breaks context. If you enqueue a Kafka message, the trace doesn't follow into the consumer unless you propagate context in the message headers.
• Background jobs lose context. Cron-triggered work has no parent trace.
• Sampling decisions need to be propagated. If the gateway decided not to trace this request, downstreams shouldn't either (otherwise you get orphan spans).
• High volume = high cost. At 10K rps, 100% sampling means storing tens of millions of spans per minute.

Applied scenarios for self-check

Further reading