Chapter 11
RAG Systems · Evaluating RAG and LLM systems
Why this matters
Without an eval set, every prompt change is vibes-based. With one, you can ship confidently. The teams that win at LLM apps in 2026 are the ones who invested in eval infrastructure before they invested in fancy retrieval.
The eval hierarchy
| Level | What it is | When to use |
|---|---|---|
| Unit / golden tests | 30-100 hand-picked Q&A pairs you know the right answer to | Every PR; CI/CD gate |
| LLM-as-judge | A bigger LLM scores your app's outputs against rubrics | Continuous integration on bigger sets |
| RAGAS metrics | Faithfulness, answer relevance, context precision/recall | RAG-specific; great for tracking drift |
| Online sampling | Score a % of real production traffic | Detecting drift, regressions; not as gate |
| Human review | Domain expert grades outputs | Calibrating the LLM-as-judge; periodic |
RAGAS — the four metrics that matter
- Faithfulness: does the answer stick to the retrieved context, or did the LLM make stuff up? (lower = hallucination)
- Answer relevance: does the answer address the question, or wander?
- Context precision: of the retrieved chunks, how many were actually relevant? (measures retrieval signal-to-noise)
- Context recall: did we retrieve everything we needed to answer? (measures retrieval coverage)
RAGAS computes these by using another LLM as judge. Quick to set up, ~$0.05 per question with GPT-4o-mini, good enough for tracking trends.
LLM-as-judge — the gotchas
- Calibrate against humans. Sample 50 outputs, have a human grade them, then have the LLM grade the same set. If they disagree on >30%, your judge prompt is broken.
- Use a different (preferably stronger) model as the judge. Don't grade GPT-4o's outputs with GPT-4o — it has a positivity bias toward its own outputs.
- Pairwise > pointwise. Asking "is A better than B?" is more reliable than "rate A 1-10".
- Position bias. LLMs prefer the first option. Randomise A/B order when comparing.
Your golden set is more valuable than your codebase
Spend a week building 200 examples that represent the actual distribution of user queries: easy, medium, hard, edge cases, adversarial. Freeze it. Every prompt change runs against it. This single artifact decides whether you ship reliably or run on prayers.
War story: team A/B tested two prompts on 10 example queries each. Prompt B "won". Shipped it. Production accuracy dropped 12%. Why? The 10 examples were drawn from a distribution that didn't represent real traffic — they had picked easy ones. Real users sent harder queries that prompt A handled better. The fix wasn't a smarter prompt; it was a better eval set.
Takeaway
You can't improve what you can't measure. Build the eval set before you optimise the prompt. The eval set is the product.