RAG Evaluation Pipelines: A Practical Tutorial for Data Teams
Build a RAG evaluation pipeline in five stages. Measure retrieval vs. generation metrics, catch silent failures, and gate regressions before deploy.
Why RAG Systems Fail Silently
Retrieval-augmented generation (RAG) systems degrade quietly. Unlike a crashed service or a failed build, a RAG pipeline that returns a confident but wrong answer throws no error. Retrieval drift, stale indexes, an upstream document format change, or a subtly reworded prompt can all erode answer quality without a single alert firing.
The mechanics of silent failure are worth spelling out, because they explain why ad hoc spot-checking does not catch them. Embedding models map text to points in a high-dimensional space, and the relationships between those points are not stable under change. Swap the embedding model, re-chunk a document, or let a source system push a new PDF template, and the nearest neighbors for a query shift—sometimes dramatically—even though every service stays green. A retrieval that once surfaced the right clause now returns three plausible-sounding chunks from a deprecated policy. The generator, doing exactly what it was asked, synthesizes a fluent answer from the wrong context.
This is the core problem: teams ship RAG without a repeatable evaluation loop and discover failures in production, usually through a frustrated user or a support ticket. By then, the regression is weeks old and no one can pinpoint which change caused it. The blast radius is also hard to bound: a single bad chunk can poison every query that retrieves it, so one ingestion bug becomes a thousand wrong answers.
The fix is not a better model or a bigger index. It is an evaluation pipeline: a disciplined, repeatable process that measures retrieval and generation separately, catches regressions before deploy, and feeds human judgment back into the loop. This tutorial walks through building one from scratch, tool-agnostic, in five stages you can implement with your existing data stack. (Related: how to design retrieval-augmented generation systems)
What a RAG Evaluation Pipeline Actually Measures
A RAG evaluation pipeline is the repeatable loop of test data → retrieval scoring → generation scoring → gating → human feedback. Each pass through the loop produces a scorecard you can compare against previous runs. The pipeline's job is to make quality measurable and regressions visible.
Retrieval Metrics vs. Generation Metrics
The single most important design decision is to evaluate retrieval and generation separately. A RAG system does two things—find relevant context and answer from it—and a combined "quality score" tells you nothing about which half broke. (See RAG architecture fundamentals)
- Retrieval metrics isolate the "find" step: recall@k (did the correct source appear in the top k?), precision@k (how much of the retrieved set was relevant?), MRR (how high did the first relevant chunk rank?), NDCG (is the ranking order sensible?), and context precision/recall (are retrieved chunks actually on-topic and sufficient?). Recall@k is the pragmatic workhorse: it is cheap to compute, easy to explain to stakeholders, and directly answers "did we even have a chance to get this right?"
- Generation metrics isolate the "answer" step: faithfulness or groundedness (is every claim supported by the retrieved context?), answer relevance (does the answer address the question?), and answer correctness (does it match the gold answer, where one exists?). These are typically scored by an LLM judge, which introduces its own reliability concerns—addressed in Stage 3.
When a metric drops, this separation tells you whether to fix your retriever, your index, or your prompt. A faithfulness drop with stable recall points at the generator or prompt; a recall drop with stable faithfulness points at ingestion or search.
The Four Failure Modes You Must Catch
Most answer-quality failures fall into one of four buckets. Your pipeline should be able to attribute any bad answer to one of them:
- Retrieval miss — the right answer exists in your corpus but never makes it into the context window.
- Context poisoning — relevant-looking but irrelevant or noisy chunks crowd out the useful ones.
- Hallucination — the answer contains claims not grounded in the retrieved context.
- Format or instruction failure — the content is correct but the output violates the required structure, tone, or schema.
These four are not exhaustive—latency timeouts, stale-but-topically-relevant documents, and multi-turn context loss are separate failure classes—but they account for the majority of answer-quality regressions and are the ones a per-stage scorecard can attribute cleanly.
If your evaluation can't tell you which of these four happened, it isn't an evaluation pipeline—it's a vibe check with extra steps.
Building the Pipeline: A Five-Stage Tutorial
The five stages below take a data team from zero to a working pipeline. Start simple; each stage adds a layer of protection.
Stage 1 — Build a Golden Evaluation Set
Everything downstream depends on your test data. Curate 50–200 representative question/answer/context triples drawn from real user queries, support tickets, and logs—not from your own imagination of what users ask. Fifty cases is enough to detect a large regression; around 200 lets you detect roughly a 10-point absolute drop in recall with conventional statistical power when you compare paired runs on the same cases. Detecting a 5-point drop reliably generally needs more cases or a paired design—don't assume 200 is a magic number. Beyond a few hundred cases, marginal returns flatten until your product surface widens.
Cover the hard cases deliberately: multi-hop questions that require combining two sources, ambiguous queries, questions about recent documents, and unanswerable queries where the correct behavior is "I don't know." Unanswerable cases are where most hallucination hides. A useful rule of thumb is that 15–25% of the golden set should be adversarial—unanswerable, near-miss, or designed to tempt the model into guessing.
Version-control the dataset alongside your code. Treat every golden-set change as a reviewed pull request, because silent edits to test data are how evaluation scores get gamed. Store each case as structured data rather than free text, so your scoring scripts can consume it programmatically. A minimal schema:
{
"id": "case_0142",
"question": "What is the refund window for enterprise plans?",
"expected_answer": "30 days from invoice date",
"gold_source_ids": ["policy_v3#refunds"],
"difficulty": "easy",
"failure_mode": "none",
"adversarial": false
}
Two operational details that bite teams later: keep stable chunk IDs across re-ingestion (otherwise gold_source_ids silently break and recall looks like it collapsed when it didn't), and deduplicate near-identical chunks so one source doesn't count three times in recall. Redact PII from logged queries before they enter the golden set. (Related: building a test dataset for LLM evaluation)
Stage 2 — Instrument Retrieval
For every query in the golden set, log the top-k chunks returned by your retriever, keyed by stable chunk ID. Then compute recall@k, precision@k, MRR, and NDCG against the gold source IDs. Run this on every change to the embedding model, chunker, index, or search configuration, and store each run's scorecard with a git SHA and config hash so you can diff runs.
The scoring loop is deliberately boring:
for case in golden_set:
retrieved = retriever.search(case["question"], k=10)
ids = [chunk.id for chunk in retrieved]
recall_at_5 = len(set(ids[:5]) & set(case["gold_source_ids"])) > 0
mrr = reciprocal_rank(ids, case["gold_source_ids"])
record(case["id"], recall_at_5, mrr, retrieved)
Pin k and report at more than one cutoff (e.g., recall@5 and recall@10); a single cutoff hides ranking regressions. Also log retrieval latency and cost per query so a quality gain that triples latency is visible in the same scorecard. Embedding the golden set on every CI run is cheap at 200 cases; at thousands of cases, sample or cache embeddings keyed by content hash.
Stage 3 — Score Generation with a Calibrated LLM Judge
With retrieval instrumented, score the answers. Faithfulness, answer relevance, and answer correctness are typically judged by an LLM, so the judge is part of your system and must be treated as such. Three rules:
- Calibrate the judge against humans. Label a held-out slice of 50–100 cases by hand, then measure agreement (Cohen's kappa for categorical judgments, correlation for scores). A judge you haven't validated is a random number generator with good manners.
- Pin the judge model and prompt version. Record both in every scorecard. Upgrading the judge model is itself a source of silent score drift—re-validate against your human slice whenever it changes.
- Control for known biases. Randomize answer order when comparing candidates (position bias), keep judge prompts short and specific (verbosity bias), and score each case independently rather than in one batch (cross-contamination).
Pass the retrieved context, the question, and the answer to the judge and ask for a structured verdict per claim where possible. Store raw judge outputs, not just aggregated scores, so you can audit disagreements later.
Stage 4 — Gate Deploys on the Scorecard
Gating turns measurement into protection. Set two kinds of thresholds:
- Absolute floors per metric (e.g., recall@5 ≥ 0.85, faithfulness ≥ 0.90), derived from your first clean baseline rather than guessed.
- Relative regression gates against the last known-good run, using a paired test on per-case pass/fail (a paired bootstrap or McNemar-style test). Block deploy when a metric regresses beyond a set margin and the paired test is significant—otherwise you'll chase noise.
Run the eval set multiple times and report mean and confidence interval, not a single number; LLM judges are stochastic. Wire the gate into CI so a failing scorecard blocks the merge, and attach the scorecard to the PR. Budget for eval cost and latency in CI (sample the golden set on every PR, run the full set nightly).
Stage 5 — Close the Loop with Human Feedback
Automated metrics catch regressions; humans catch the cases your metrics don't model. Route production failures—thumbs-down, escalations, low-confidence answers—into a review queue. Reviewers label the failure mode (one of the four buckets), and confirmed failures become candidate golden-set cases. This is how the golden set stays representative as your product and corpus evolve, and how you discover metric blind spots. Keep the review cadence fixed (e.g., weekly) so the loop doesn't stall when the team gets busy.
The pipeline is complete when a bad answer in production can be traced to a stage, attributed to a failure mode, reproduced in the golden set, and prevented from shipping again.