LLM Engineering · Cost, latency, and reliability

Why this matters

Demos run on $50/month. Production at 1K daily users runs on $5K-$50K/month depending on how careful you are. The difference is mostly engineering, not which model you picked.

Token economics

Output tokens are 4× more expensive than input. Short, structured outputs save more money than people realise. "Respond with one word" can shave 60% off a high-volume workload.

Four cost levers that compound

1. Model routing: cheap model first, escalate only if needed. 50-80% savings on mixed workloads.
2. Prompt caching: providers (OpenAI, Anthropic, Gemini) cache the prefix of a prompt if it's the same. Put your stable system prompt + examples first, variables last. 50-90% reduction on cached portion.
3. Batch inference: OpenAI Batch API costs 50% of real-time. Use for nightly evaluations, backfills, embedding generation.
4. Open-source for high-volume: at >500K calls/month, hosting Llama 3.1 70B on Groq/Together/Fireworks is often cheaper per token than GPT-4o-mini, with comparable quality on most tasks.

The latency budget

Time-to-first-token (TTFT) is what users perceive. Total-response-time matters for non-streaming integrations. A back-of-envelope:

• Network RTT to provider: 50-200ms
• TTFT: ~300ms-1s depending on model + load
• Output rate: 30-100 tokens/sec for frontier models, 200+ for Groq
• 500-token response: ~5-15s wall time

If perceived latency matters: stream. If you can't stream (e.g., you need to validate the full structured output): pick a smaller model, or split into multiple cheaper calls.

Reliability patterns from chapter 1, applied

• Timeouts on every call. 30 seconds is generous.
• Retry with exponential backoff + jitter on 429, 5xx, timeouts.
• Circuit breaker on provider failures — fall back to a secondary provider or a degraded response.
• Idempotency keys on mutations the LLM triggers (especially payment-adjacent).
• Token budgets per request: max_tokens=N. Runaway generations have run companies dry.