Production · Observability and tracing

Why this matters

Traditional APM (Datadog, New Relic) tells you "the request took 4 seconds". For LLM apps that's the start, not the end. Which prompt? Which tool calls? What did the model say? Was the answer good? Standard logs are blind to all of it.

The three pillars — LLM style

• Tracing: the full call graph for one user request — every LLM call, every tool, with inputs and outputs.
• Metrics: counters and percentiles (p50/p95/p99 latency, token spend per endpoint, error rates by failure type).
• Evals: online sampling — score a percentage of production responses against rubrics, catch quality regressions early.

The tools in 2026

Trace anatomy — what to capture

Every span should carry:

• session_id + user_id — group spans into user journeys
• prompt_hash — fingerprint the system prompt version
• model, provider, temperature, max_tokens
• input_tokens, output_tokens, cost_usd
• latency_ms (time to first token + total)
• tool_calls — name, args, result, error if any
• finish_reason — stop / length / content_filter / tool_calls

Online evaluation sampling

You can't eval every request (too expensive). Sample 1-5% of production traffic, run it through an LLM-judge with your rubric, store the score. Alert when the rolling-window faithfulness drops more than 2σ from baseline.

# Pseudo-code: post-response hook
if random.random() < 0.02:                    # 2% sampling
    eval_result = await judge_llm.evaluate(
        prompt=stored_prompt,
        response=stored_response,
        rubric=PRODUCTION_RUBRIC,
    )
    await metrics.record_eval(
        eval_score=eval_result.score,
        prompt_hash=prompt_hash,
    )

Cost dashboards (the one your CFO will ask about)

Top-line: $/day total. Drill down: $/day by endpoint, $/day by user, $/day by model. Alert thresholds: 2× rolling-7-day average. The day someone deploys a 10× cost regression, you want to know within an hour, not a week.