Cost engineering

Why this matters

Cost is a design dimension, not an afterthought. Junior engineers think about correctness; senior engineers add performance; Staff engineers add cost as a first-class concern. Production systems are evaluated on cost-per-user, cost-per-request, cost-per-transaction — not just whether they work.

Cost engineering is also where you most clearly demonstrate Staff judgment. Building something for $X/month when it could be built for $X/10 with a different architecture is a Staff-level call. The math matters.

The cloud cost mental model

Major cost categories on any cloud provider:

For a typical SaaS, the cost breakdown is usually:

• Compute: 30-50%
• Managed databases: 20-40%
• Network egress: 10-30% (often surprising)
• Storage: 5-15%
• Everything else: 5-15%

Egress — the silent killer

Egress is bandwidth leaving the cloud (to internet, to another region, to another provider).

AWS egress to internet: ~$0.09/GB. At 1 TB/day of traffic: $90/day, $2,700/month. Seems small. But:

• Cross-region egress: same $0.02-0.08/GB.
• If you're shuffling 100 TB/month between regions: $5,000-8,000/month just in transfer fees.
• If your service serves 10 PB/month of video: $900K/month.

This is why Cloudflare exists (free egress relative to AWS), why CloudFront is much cheaper than direct S3 serving, why Discord built their own CDN, why Backblaze and competitors have flat-rate egress.

Capacity vs cost vs latency triangle

You can pick two:

• Low cost + low latency: requires careful capacity planning. Over-provisioning is wasteful; under-provisioning hurts latency.
• Low cost + high capacity: under-provision and let latency suffer.
• Low latency + high capacity: over-provision generously.

Most Staff engineers operate on the first axis: tight capacity planning to balance cost and latency.

Per-request / per-user cost

The number every Staff engineer should be able to compute for their service:

cost_per_request = (total_monthly_infra_cost) / (monthly_requests)
cost_per_user = (total_monthly_infra_cost) / (monthly_active_users)

Knowing this:

• You can evaluate "is this feature worth building?" by comparing its cost impact to its revenue impact.
• You can prioritise optimisations: 80% of cost usually comes from 20% of code paths.
• You can defend the architecture to leadership: "this costs $X per user, our revenue is $Y per user, margin is healthy."

Worked example: LLM-backed feature

A chat feature uses Claude Sonnet 4 at $3/M input tokens, $15/M output tokens. Average chat turn: 2K input tokens (cached prefix + recent history), 500 output tokens. So per turn:

cost_per_turn = (2000 × $3 + 500 × $15) / 1M = $0.0135 per turn

With caching at ~50% discount on cached input: ~$0.008 per turn.

If your free tier averages 20 turns/user/month: $0.16/user/month for LLM. At 1M MAU, that's $160K/month. Your business model must support this.

If switching to GPT-4o-mini ($0.15/M input, $0.60/M output): per turn $0.0006. Per user: $0.012/month. At 1M MAU: $12K/month. 10× cost reduction by picking a different model.

This is the kind of math that decides product viability.

Reserved vs spot vs on-demand

A well-tuned cloud bill: ~70% reserved/savings plans (baseline) + ~20% spot (batch / stateless scale-out) + ~10% on-demand (spike capacity). Typical savings: 40-50% vs all-on-demand.

Auto-scaling economics

Auto-scaling sounds free but isn't. Each scale-out event has a delay (new instance boot time, warm-up). Scale-in is also fraught (don't kill instances with in-flight requests).

For predictable load (daily peaks): pre-scale before peak. Use scheduled scaling, not reactive.

For unpredictable load: react fast enough to absorb spikes, but not so fast that you flap (scale up, then scale down, then up).

For latency-sensitive: don't auto-scale on average load; scale on p99 latency or queue depth. Average can be fine while tails are awful.

When caching is more expensive than the DB it caches

A surprising production reality. Run the numbers before adding cache.

Example: a 16GB Redis cluster costs ~$300-500/month. The DB queries it caches cost maybe $0.0001 each. Cache pays off if you're hitting cache > 3-5M times/month and DB queries would otherwise be that many.

For low-traffic services with expensive caches: the DB hit it would have been cheaper. Reason about hit rate × DB cost × volume vs cache cost.

FinOps practices

FinOps = bringing financial accountability to cloud spending.

• Cost dashboards by team/service: every team sees their own spend, monthly.
• Tagging discipline: every resource tagged with owner, environment, service. Untagged resources flagged.
• Anomaly detection: alert on spend spikes (5x normal in a day).
• Periodic optimisation reviews: quarterly look at top spenders, opportunities.
• Reserved capacity planning: forecast baseline, buy commitments to match.
• Idle resource cleanup: dev environments, test clusters, forgotten resources.

Applied scenarios

Further reading