Production · Cost engineering at scale

Why this matters

A working LLM app's first cost shock arrives at ~5,000 daily users — that's when "interesting bill" becomes "alarming bill". Teams that survive instrument cost early and treat it like latency: it's a budget you spend on capability.

The cost levers, in order of impact

1. Don't call the LLM when you don't need to. Cache exact-match responses, rule-based shortcuts for common queries.
2. Model routing: tier requests by complexity — Haiku/4o-mini handles 80% trivially; Sonnet/4o gets the hard 20%.
3. Prompt caching: providers cache stable prefixes. Put system prompt + examples first, variables last. 50-90% savings on cached portion.
4. Truncate outputs: ask for "one word" if you need one word. Output tokens are 4× input cost.
5. Batch inference: OpenAI batch API at 50% cost; great for nightly evals, backfills, embedding generation.
6. Open-source crossover: at >1M calls/month on a stable task, hosting Llama on Groq/Together is often cheaper per call than any frontier API.
7. Eliminate the call: if "summarise this 5-word title" is being LLM'd, you're cooked. Audit your highest-traffic paths.

Building a per-user cost model

Track cost per user, not just total. Knowing your 95th-percentile user costs $0.40/day is the difference between "we can afford a free tier" and "the free tier kills us at 50K users". Tag every call with user_id in your observability stack; aggregate nightly.

When self-hosting becomes cheaper

Rough rule for 2026: on a stable, latency-sensitive workload running 24/7, self-hosting a 70B model on Together AI or Fireworks crosses the API breakeven around 500K-1M output tokens/day. Below that, APIs win. Above it, do the math properly with your actual workload.