LLM Engineering · Provider landscape and tradeoffs
Why this matters
The "LLM" in your app is a vendor-locked dependency. Choosing one without understanding the tradeoffs costs you on day 60 when the cheaper alternative would have saved $40K/month, or on day 200 when the provider hikes prices and you can't switch because your prompts are tied to one quirk.
The 2026 landscape
| Provider | Flagship | Strengths | Watch out for |
|---|---|---|---|
| OpenAI | GPT-4, GPT-4o | Reliability, ecosystem, function calling maturity | Highest pricing tier; aggressive rate limits early |
| Anthropic | Claude 3.5/4 | Long context (200K+), nuanced instruction-following, structured-thinking modes | Tool use API differs from OpenAI's — porting code requires rewrites |
| Gemini 1.5/2.0 | 1M+ token context, native multimodal (image/video/audio) | Tool calling is newer; eval consistency is still maturing | |
| Open source | Llama 3.x, Mistral, Qwen | Self-host or cheap inference (Groq, Together, Fireworks); no vendor lock | Reasoning ceiling lower than frontier closed models for now |
| Specialised | Cohere Command (RAG), Mistral Codestral (code), Voyage (embeddings) | Best-in-class for specific tasks | Smaller ecosystem, fewer SDKs |
Picking a default
For most new projects in 2026: start with Claude 3.5 Sonnet or GPT-4o. Both are reliable, well-documented, support function calling and structured output, and have strong reasoning. Don't waste a week comparison-shopping on day 1 — pick one, ship the prototype, then benchmark alternatives once you have evals.
Abstract the provider on day one
Even if you only use one provider, write a thin adapter so you can swap. Three-line interface, two implementations behind it. Saves a multi-week rewrite when you decide to A/B providers six months in.
class LLM(Protocol):
async def complete(self, messages, *, model, tools=None, temperature=0.0) -> Response: ...
async def stream(self, messages, *, model, tools=None) -> AsyncIterator[Chunk]: ...
The router pattern (cheap → expensive)
Production apps rarely use one model. They use a router: a cheap fast model handles the 80% easy cases; an expensive smart model handles the 20% hard ones. The classifier is itself an LLM call (cheap one). Done right, this halves your bill without quality loss.
flowchart LR
U[User query] --> C{Classifier
small + cheap}
C -->|trivial| F[Haiku / GPT-4o-mini]
C -->|complex| S[Sonnet / GPT-4o]
C -->|code| K[Specialist coder]
F --> O[Response]
S --> O
K --> O
Takeaway
Pick a model the way you'd pick a database — by reading the docs, benchmarking on your workload, and abstracting it behind an interface so you can switch. Anyone betting their entire architecture on one provider's quirks is buying technical debt at a discount.