Chapter 15
AI Agents · Reliability — failure modes and guardrails
Why this matters
Demo agents work because the path was rehearsed. Production agents fail in eight reliably-bad ways. Knowing them is the difference between a system that 99% works and one that 80% works.
The eight failure modes
| Failure | Symptom | Defence |
|---|---|---|
| Infinite loop | Agent calls same tool repeatedly with same args | Iteration cap + duplicate-call detector |
| Tool hallucination | Agent invents a tool name that doesn't exist | Server-side validation; return structured error; LLM recovers |
| Wrong tool | Picks similar-sounding tool that doesn't fit | Better tool names + descriptions (chapter 6) |
| Argument hallucination | Calls tool with made-up IDs/values | Strict schema validation; require explicit "search first" for unknown IDs |
| Premature stopping | Returns answer before doing required work | System prompt: "Before answering, verify..." + critic agent |
| Context drift | Forgets early instructions after 10+ turns | Re-inject system prompt periodically; bound iterations |
| Silent failure | Tool returns error; agent confabulates over it | Always observe tool output; assert success in system prompt |
| Cost runaway | Agent works fine but burns $50 per task | Per-request token + tool-call budget; circuit breaker |
Defence in depth — the layers
flowchart TB
U[User request] --> I[Input validation
length, content moderation]
I --> P[Prompt scaffolding
system rules + few-shot]
P --> L[LLM call]
L --> S[Schema validation
did we get expected shape?]
S --> T{Tool call?}
T -->|yes| TC[Tool exec
with try-catch + retry]
TC --> L
T -->|no| O[Output filter
PII redaction, content moderation]
O --> User
Self-correction patterns that work
- Verify-then-answer: agent must produce a "verification" step that explicitly checks evidence supports the answer.
- Critic loop: a second agent reviews the first agent's output against rules and sends back specific corrections.
- Confidence scoring: model rates its own confidence; below threshold → fall back to "I don't know" or escalate to human.
- Test-driven generation: for code/structured output, generate alongside test cases; reject and regenerate on test failure.
War story: agentic customer-support bot was confidently wrong 12% of the time, citing real-looking but fake order numbers. Added a verifier pass: "before responding, the order ID you cited must come from the tool output above". Wrong-order rate dropped to 0.4%. Cost: one extra LLM call per response (~$0.001). Worth it.
Production reliability checklist
- Iteration cap set + tested (find the cap by running 50 hard inputs)
- Every tool wrapped in try/except returning structured error
- Token budget per request; circuit breaker on N consecutive expensive runs
- Telemetry on every step: which tool, which args, latency, cost
- Golden eval set runs on every deploy + nightly
- Online sampling: 1% of production traffic gets LLM-judge eval
- Hallucination metric tracked over time; alert on drift
- "I don't know" path: agent can refuse confidently when underspecified