Production · Serving LLM applications
Why this matters
Demo LLM apps live in a notebook. Production ones face concurrent users, network blips, intermittent provider failures, streaming requirements, and a need to recover from crashes. Getting the serving layer right separates "interesting demo" from "actual product".
Four deployment shapes — pick one
| Shape | When it fits | Caveat |
|---|---|---|
| Stateless API (FastAPI on Render/Fly/Railway) | Most LLM apps with external state | Scale by running more instances; LLM I/O dominates so cheap |
| Edge functions (Vercel, Cloudflare Workers) | Front-end-adjacent, low cold-start tolerance | Timeout limits (Vercel 60s for Pro); streaming works but cold-starts add latency |
| Job queue + workers (Celery, BullMQ, Inngest) | Long-running agents (1+ minute), background pipelines | Adds operational complexity; need result polling or webhook |
| BentoML / Modal / Runpod | Self-hosted LLM inference; GPU-backed | Real ops; pays off only when API cost outweighs infra cost |
Streaming — the table stakes
First-token latency is what users perceive. Stream from the model to the user; non-streamed responses feel broken in 2026.
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
app = FastAPI()
@app.post("/chat")
async def chat(req: ChatRequest):
async def gen():
async for chunk in llm.stream(req.messages):
yield f"data: {json.dumps({'token': chunk.text})}\n\n"
yield "data: [DONE]\n\n"
return StreamingResponse(gen(), media_type="text/event-stream")
Server-Sent Events (SSE) is the default; WebSockets when you need bidirectional comms (agent tool callbacks streaming back, interruptible generation).
Queue architecture for long-running agents
flowchart LR
C[Client] --> API[/POST /jobs/]
API --> Q[(Job queue
Redis/SQS)]
API --> C2[Return job_id immediately]
Q --> W[Agent worker]
W --> M[(Status store)]
C --> P[/GET /jobs/:id/]
P --> M
W -->|on complete| WH[Webhook to client]
Anything that takes > 30 seconds belongs on a queue. Returning a job ID and letting the client poll or receive a webhook beats holding open an HTTP connection for 2 minutes.
Rate limiting and per-user quotas
Three layers:
- Per-IP / per-user request rate (Redis token bucket): protects from abuse and runaway clients.
- Per-user daily token quota: prevents one user from burning your budget. Tracked in the same Redis or your DB.
- Per-tier model access: free users on smaller models, paid on frontier. Routed at the entry point.
Circuit breakers for upstream providers
OpenAI/Anthropic occasionally have multi-hour outages. Production apps should not stop working. Pattern: primary provider + fallback provider, circuit breaker on the primary that flips to fallback after N consecutive failures, auto-recovery after cooldown.