Capacity planning & queueing theory

Why this matters

Most engineers cannot size systems. They guess. They over-provision (waste money) or under-provision (cause outages). Staff engineers reflexively quantify: throughput, latency, headroom, cost.

The underlying math — queueing theory — was developed in the 1900s for telephone networks. It applies directly to modern systems: web servers, databases, queues, CPUs. Internalising even a small subset of it gives you a quantitative model that beats intuition.

This topic is also where you understand why systems behave the way they do under load. Why does latency explode nonlinearly when utilisation hits 80%? Why does adding more workers sometimes make things worse? Why does p99 latency get much worse than median? The answers are in the math.

Little's Law — the most useful formula

Little's Law states:

L = λ × W

where:
  L = average number of items in the system
  λ = average arrival rate (items per unit time)
  W = average time spent in the system (per item)

The remarkable thing about Little's Law is its generality: it holds for any system in steady state, regardless of the distribution of arrival times or service times. As long as the system is stable (not growing infinitely), this relationship holds.

Applied examples

Web server: a service processes 1,000 requests/second (λ). Each request takes on average 50ms (W = 0.05s). How many requests are "in flight" at any moment?

L = 1,000 × 0.05 = 50 requests in flight on average

This tells you: you need a connection pool / worker pool of at least 50, and probably 2-3× that for buffer.

Database connection pool: you have an app server doing 500 queries/sec (λ), each query takes 10ms (W = 0.01s). Average concurrent queries:

L = 500 × 0.01 = 5 concurrent queries

You need at least 5 connections. With burst, maybe 20.

Kafka partition: a partition is read at 100 msg/sec by a consumer that takes 50ms per message. Concurrent messages "in flight" per partition:

L = 100 × 0.05 = 5

If your consumer can only process one message at a time (single-threaded per partition), and processing takes 50ms, your max throughput is 1 / 0.05 = 20 msg/sec. To hit 100 msg/sec, you need 5 partitions (or parallel consumers per partition).

Inverting Little's Law

Often the question is: given my system's constraint (L), how much can I scale?

λ = L / W

If my pool size is 100 (L=100 max) and each request takes 50ms (W=0.05s):
λ = 100 / 0.05 = 2,000 requests/sec max throughput

Or: if I want to hit 5,000 req/sec with 50ms latency, I need a pool of at least:

L = 5,000 × 0.05 = 250

Queueing theory — M/M/1 model

The M/M/1 queue is the simplest model: a single server (1), Markov (memoryless, Poisson) arrivals (M), Markov service times (M). Real systems are more complex but the math gives you intuition.

Parameters:

• λ = arrival rate
• μ = service rate (1 / average service time)
• ρ = utilisation = λ / μ (the fraction of time the server is busy)

The killer result: average time in system (waiting + service) is:

W = 1 / (μ − λ) = 1 / μ × 1 / (1 − ρ)

The 1 / (1 − ρ) factor is the punchline. As ρ approaches 1 (server fully utilised), W goes to infinity. This is the utilisation curve — the most important graph in system design.

The utilisation curve, with numbers

xychart-beta
    title "Queue wait time vs utilisation (M/M/1)"
    x-axis "Utilisation %" [10, 30, 50, 60, 70, 80, 85, 90, 95, 99]
    y-axis "Wait time multiplier" 0 --> 100
    line [0.1, 0.4, 1.0, 1.5, 2.3, 4, 5.7, 9, 19, 99]

Suppose your server can process requests in 100ms (μ = 10 req/sec). What does latency look like at different load levels?

At 50% utilisation, you're 2× the unloaded latency. At 80%, you're 5×. At 95%, you're 20×. At 99%, you're 100×.

(Note: this is the M/M/1 model. Real systems with multiple servers per queue, batching, or different service-time distributions behave differently — but the qualitative shape is similar. Many-server queues are more forgiving but the same hockey-stick exists.)

Head-of-line blocking

When work units must be processed in order, one slow unit blocks everything behind it. The single-server queue is the simplest example.

Examples in real systems

• HTTP/1.1 connection: requests queue on a single TCP connection; one slow response holds up the rest. HTTP/2 introduced multiplexing to fix this.
• Kafka partition: messages within a partition are consumed in order. One slow consumer call blocks subsequent messages.
• Database connection: queries on a single connection are serialised. One slow query blocks others. Hence connection pools.
• Single-threaded event loop: Node.js, Redis. One slow operation blocks the entire event loop.

Mitigations

• Parallelism: more workers, more connections, more partitions.
• Priority queues: don't make critical work wait behind low-priority.
• Bounded queues + load shedding: prefer rejecting work to letting queues grow unbounded (latency goes to infinity, see above).
• Timeouts: kill slow work so the queue can move.

Tail latency amplification

Suppose each backend server has 1% probability of being slow (above some threshold). You call one backend → 1% chance of slow response.

But suppose your request fans out to 100 backends, all of which must respond before you can reply (e.g., a federated search). What's the probability that at least one is slow?

P(at least one slow) = 1 − (0.99)^100 ≈ 63%

So 63% of fanout requests will see at least one slow backend, even though only 1% of individual backend calls are slow. Tail latency amplifies with fanout.

This is the central insight in Jeff Dean and Luiz Barroso's The Tail at Scale paper (2013). Required reading.

Mitigations

• Hedged requests: after some delay (e.g., p95 threshold), fire a duplicate request to a different backend. Take whichever returns first. Costs 2-5% extra backend load; cuts p99 dramatically.
• Tied requests: send the request to multiple backends with priority; the first to start cancels the others.
• Cancel on response: when one backend returns, cancel the others (avoid wasted work).
• Reduce fanout: not always possible, but often the cheapest fix.

Capacity-planning calculation walkthrough

You're designing an API service. Requirements:

• Peak traffic: 10,000 requests/second
• Average request: 5KB request body, 50KB response body
• Average request processing time: 80ms (50ms backend calls + 30ms compute)
• p99 latency target: 300ms

Step 1: Concurrency. By Little's Law, average concurrent requests = 10,000 × 0.08 = 800. Plus burst: target 1,500 concurrent capacity.

Step 2: CPU. 30ms compute per request × 10,000 req/s = 300 CPU-seconds per second. So 300 cores running at 100% utilisation — but you target 50% utilisation, so 600 cores. Round up to 700.

Step 3: Network. Request: 5KB × 10,000 = 50 MB/s ingress. Response: 50KB × 10,000 = 500 MB/s egress. 500 MB/s = 4 Gbps. Need 10 Gbps NIC capacity, with headroom.

Step 4: Memory. 1,500 concurrent × ~200KB working memory per request = 300 MB. Negligible.

Step 5: Instances. If each instance is 8 CPU / 16GB / 1 Gbps:

• CPU: 700 cores ÷ 8 = 88 instances
• Network: 500 MB/s ÷ 100 MB/s per instance = 50 instances
• Memory: not the binding constraint
• Pick the max: 88 instances

Step 6: Headroom for AZ failure. Spread across 3 AZs; design for losing one. Need 88 × 1.5 = 132 instances total.

This is the kind of calculation Staff engineers do reflexively. The numbers might be approximations, but the discipline of going through each resource explicitly prevents "let's just put it on a couple of c5.xlarge and see."

Applied scenarios

Further reading