Failure handling: timeouts, retries, idempotency

Why this matters

Production systems fail in ways textbooks don't cover. Books talk about node crashes; production fails because of slow networks, partial degradation, cascading retries, half-dead processes, GC pauses, and a million small things. Senior engineers separate themselves from juniors here.

The AWS Builder's Library is the gold standard for this material. Read every post.

Timeouts — the most under-set value

Three timeouts every network call needs:

1. Connection timeout: max time to establish the TCP connection. Should be aggressive — under 1 second. If a downstream is unreachable, you want to know quickly.
2. Read timeout: max time waiting for response after request is sent. Size to the p99 of downstream's operation.
3. Total deadline: end-to-end budget for the request including retries.

Universal rule: every network call needs an explicit timeout. Default of "infinite" is a bug. Make your timeout shorter than the upstream caller's timeout (so you fail fast and they can retry intelligently). Pass the remaining deadline to downstreams via headers (X-Deadline, gRPC deadlines).

Retries with exponential backoff and jitter

The standard formula:

delay = min(max_cap, base × 2^attempt) × random(0.5, 1.0)

Why backoff: if a downstream is overloaded, retrying immediately makes it worse. Give it breathing room.

Why exponential: catches transient failures fast; backs off aggressively for persistent ones.

Why jitter: without it, all clients retry at exactly the same time after an outage. The downstream comes up, gets slammed by synchronised retries, falls over again. Jitter spreads the herd.

Jitter strategies

Read Marc Brooker's "Exponential Backoff and Jitter" for the data.

Retry budgets — more important than retry count

The single most-underappreciated concept in retry design.

• Per-request budget: max N retries within a total deadline. Common: 3 retries within 30 seconds.
• Per-process budget: retries shouldn't exceed ~10% of total request volume. If they do, the downstream is broken and you're self-DDoSing.

Without retry budgets, a downstream outage causes traffic amplification: every failure → 4 requests (1 original + 3 retries) → 4× load on downstream → makes the outage worse. Retry storms are a real production failure mode.

When NOT to retry

• 4xx errors (except 429): client problem; retry won't help.
• Non-idempotent operations without idempotency keys: you'll double-charge users.
• Already past deadline: the caller won't get the response anyway.
• Circuit breaker is open: fail fast and let recovery happen.

Idempotency in depth

An operation is idempotent if performing it N times has the same effect as performing it once. set(x, 5) is idempotent; increment(x) is not. Idempotency is what makes retries safe.

Pattern 1: Idempotency keys (Stripe style)

Client generates a UUID for each logical operation. Server stores results keyed by UUID for a window (typically 24 hours). Duplicate requests return the cached result.

POST /v1/charges
Idempotency-Key: 7f2c1a-9b4d-4e8a-...
{ "amount": 1000, "currency": "INR", "source": "tok_..." }

Server pseudocode:
result = SELECT response FROM idempotency_keys WHERE key = $1 AND expires_at > now();
IF result IS NOT NULL:
  RETURN result;  // cached response

// New request — claim the key atomically
claimed = INSERT INTO idempotency_keys (key, request_hash, status)
          VALUES ($1, $2, 'pending')
          ON CONFLICT (key) DO NOTHING
          RETURNING id;

IF claimed IS NULL:
  // Lost the race — another request is processing this key right now
  // Either wait (poll) or return a "conflict, please retry" error

// We won — process the request
response = process_charge();
UPDATE idempotency_keys SET response = $1, status = 'complete' WHERE key = $2;
RETURN response;

Critical correctness detail: the ON CONFLICT DO NOTHING RETURNING pattern is what makes two concurrent first-time requests safe — only one wins the claim. A naive "SELECT, if not found, INSERT" pattern is broken under concurrency.

Pattern 2: Naturally idempotent operations

Some operations are intrinsically safe to retry:

• PUT /resource/123 {state}: setting absolute state is idempotent. Two identical PUTs = one effect.
• DELETE /resource/123: deleting twice = same as deleting once.
• UPDATE balance SET amount = 100 WHERE id = X: absolute write. Idempotent.
• UPDATE balance SET amount = amount + 100 WHERE id = X: relative write. NOT idempotent — retrying doubles the increment.

Design pattern: prefer absolute values over deltas whenever possible. If you must use deltas (counters, accumulators), pair with idempotency keys or use deduplication at the consumer.

Pattern 3: Tombstones / processed-events tables

For consumers of an event stream: maintain a "processed" table. Before processing an event, check if event_id has been seen. After processing, mark it seen.

BEGIN;
  // Inside the same transaction as the side effect
  INSERT INTO processed_events (event_id) VALUES ($1)
  ON CONFLICT (event_id) DO NOTHING
  RETURNING event_id;
  
  // If RETURNING is null, event was already processed — skip
  // Else: do the side effect
  
COMMIT;

Circuit breakers

stateDiagram-v2
    [*] --> Closed
    Closed --> Closed: success
    Closed --> Open: failure rate > threshold
    Open --> Open: requests fail-fast
    Open --> HalfOpen: cooldown elapsed (e.g. 30s)
    HalfOpen --> Closed: trial request succeeds
    HalfOpen --> Open: trial request fails

Stop hammering a failing downstream. Three states with hysteresis (preventing flapping).

CLOSED (normal)
   |
   | failure rate > threshold for N seconds with min M requests
   ↓
OPEN (fail-fast — don't even try downstream)
   |
   | after cooldown_seconds
   ↓
HALF-OPEN (let one probe through)
   |
   | probe succeeds → CLOSED       | probe fails → OPEN (reset cooldown)
   ↓                                 ↓
CLOSED                              OPEN

Typical production parameters:

• Failure threshold: 50% errors over 30s window, minimum 20 requests.
• Open duration: 30s.
• Half-open probe: 1 request at a time.

Where to put the breaker: client-side, per-downstream. Per-service might be too coarse if one endpoint of a service is broken but others are fine. Sidecars like Envoy can do this centrally.

Why circuit breakers matter: without them, a slow downstream causes thread/goroutine accumulation in your service → memory pressure → garbage collection → cascading failure across your whole fleet. The breaker turns "slow downstream" into "fast failure," letting your service stay healthy and serve degraded responses.

Bulkheads

Isolate failure domains. If your service calls downstreams A, B, and C, give each its own thread pool and connection pool.

[Your Service]
   ├── HTTP client pool A (for downstream B): 50 conns, 50 threads
   ├── HTTP client pool C (for downstream D): 50 conns, 50 threads
   └── DB connection pool: 30 conns

When downstream D becomes slow, only its pool saturates. Pool A (calls to B) is unaffected. Your service can still serve traffic that only depends on B.

Real production wins from bulkheads:

• Per-tenant resource quotas (one tenant's outage doesn't poison others).
• Per-endpoint connection pools (slow endpoint doesn't kill others).
• Separate thread pools for batch vs interactive workloads.
• Separate processes for risky operations (parsing untrusted input, calling third parties).

Load shedding

When overloaded, drop requests strategically rather than crash. Priority-aware:

The principle: a 50% successful service is better than a 0% crashed service.

Implementation: a request priority header from upstream; a load-shedding layer (often at the edge / load balancer) that rejects requests above a configured priority when CPU/memory/queue depth crosses thresholds.

Hedged requests

Send the same request to two backends in parallel; take the first response that arrives. Hedges against tail latency — if one replica is slow, the other isn't.

Cost: 2× backend load. Only worth it for tail-sensitive workloads. Implement smartly: only hedge if first request takes longer than p95 of expected latency, so you don't double load for fast cases.

Used by: BigTable, Spanner, MongoDB drivers, gRPC retry policies.

The full pattern, end to end

1. Client makes request with:
   - Idempotency key (generated client-side)
   - Deadline (max total time)
   - Request ID (for tracing)

2. Service receives request. Checks circuit breaker for downstream A.
   - If OPEN: return cached or fallback response immediately.

3. Try the call to downstream:
   - Connection timeout: 1s
   - Read timeout: 3s
   - Pass deadline header to downstream

4. On failure:
   - Decide if retryable (5xx yes, 4xx no, timeout maybe)
   - Check retry budget: per-request (3 max) AND per-process (under 10% retry rate)
   - Check deadline (do we have time left?)
   - If yes to all: retry with backoff + jitter
   - Else: record failure on circuit breaker, return typed error

5. On success:
   - Cache response by idempotency key
   - Reset breaker failure count
   - Emit metrics

6. Emit metrics throughout:
   - Latency histogram
   - Success/failure count
   - Retry count
   - Circuit breaker state
   - Deadline-budget-remaining at each step

Interview drills

CREATE TABLE outbox (
  id BIGSERIAL PRIMARY KEY,
  aggregate_id UUID,
  event_type TEXT,
  payload JSONB,
  partition_key TEXT,
  created_at TIMESTAMPTZ DEFAULT now(),
  processed_at TIMESTAMPTZ
);
CREATE INDEX ON outbox (partition_key, id) WHERE processed_at IS NULL;

SELECT * FROM outbox
WHERE processed_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;

Red flags

• Doesn't set explicit timeouts.
• "Just retry 3 times" without backoff or jitter.
• Suggests retrying 4xx errors.
• Implements idempotency by reading then writing (lost-update race).
• Conflates circuit breakers with rate limiters.
• Doesn't mention deadline propagation.