Evaluating RAG and Agents Together: A 2026 Data Science Playbook for Reliable AI Systems
Learn how to evaluate RAG and agents together in one pipeline. A 2026 playbook for joint evals, trajectory scoring, and reliable AI systems in production.
How Do You Evaluate RAG and Agents Together?
Joint evaluation means scoring retrieval quality and agent trajectory quality against the same task-level ground truth, in one pipeline. Instead of running a RAG eval suite and an agent eval suite in parallel and reconciling them by hand, you define each task once — expected answer, expected evidence, expected tool path — and instrument every layer of the system against it.
The payoff is structural, not just cosmetic. When both surfaces share one task definition and one trace format, a wrong answer can be traced to a specific layer in a single trace diff instead of days of cross-dashboard archaeology. You stop arguing about whose metric is right and start reading one artifact that shows where the system diverged from the expected path. If your retrieval metrics and your agent metrics live in different dashboards, you do not have an evaluation strategy. You have two opinions.
Why Separate RAG and Agent Evaluation Breaks in Production
The classic split — RAG teams own retrieval metrics, agent teams own task success — fails the moment the two systems share a runtime.
RAG-only evals cannot see a correct retrieval that the agent then misuses: the right passage is in context window position 9, the agent ignores it, calls a stale tool, and hallucinates. Every retrieval metric reports green. Agent-only evals cannot see the inverse: the agent succeeded because it got lucky, routing around garbage context with a well-chosen tool call. Task success reports green, and you ship a system that will fail the next time the query is phrased differently.
Compounding Error: The Math That Kills Isolated Metrics
Assume each step in an agent path is 95% reliable — a generous number. A six-step path lands at roughly 74% end-to-end success (0.95⁶ ≈ 0.735). At ten steps, you are at 60%.
This is a simplification, and it is worth stating the assumption: the product model treats step failures as independent. In practice they are correlated. A broken tool schema fails every step that touches it (worse than the model predicts); a strong retriever makes downstream reasoning easier (better than the model predicts). Use the product as a lower-bound intuition, not a forecast — the point stands either way. Retrieval is usually one of those multiplicative terms. If you evaluate it in isolation at 95% and feel good about it, you have missed that it is one of several factors in a product that is quietly degrading.
A 95% retrieval score inside a 74% task success rate is not a passing grade. It is a rounding error in a failing system.
The boundary has also collapsed architecturally. In 2026 stacks, the "retrieval layer" is frequently a model making decisions: multi-hop search, tool-augmented lookup, query rewriting, and sub-agent delegation. There is often no clean seam to evaluate on either side of.
The Two-Layer Failure Model: Retrieval vs. Reasoning
Split every failure into two layers before you debug anything.
- Layer A — Retrieval failures. The evidence needed to answer correctly was never in context, or was ranked below the effective attention window. Causes: chunking strategy, embedding drift, index staleness, query rewriting errors, reranker miscalibration.
- Layer B — Reasoning and orchestration failures. The evidence was present and correct, but the agent ignored it, misordered steps, called the wrong tool, passed bad arguments, or terminated early.
The diagnostic value is in the cross-tab:
| Reasoning good | Reasoning bad | |
|---|---|---|
| Retrieval good | Release candidate | Prompt / tool-schema problem |
| Retrieval bad | Indexing / search problem | Task-definition problem |
- Retrieval good + reasoning bad → fix prompts and tool schemas.
- Retrieval bad + reasoning good → fix indexing and search.
- Both bad → your golden set is probably asking for something the system was never designed to do. Re-examine task definition before touching the model.
- Both good → the only cell that earns a release.
Where Single-Layer Metrics Mislead You
Concrete false-pass scenarios you will hit within a month of shipping:
- Faithfulness passes on empty or trivial context. If the retrieved set is small or the answer is short, groundedness scores inflate. A system that retrieves nothing and answers from parametric memory can score well on "is the answer supported by context" when the judge is lenient. Guardrail: never report faithfulness without the paired context recall, and treat faithfulness as undefined (not passing) when retrieved context is below a minimum token threshold.
- Context recall passes, answer fails. You retrieved the needle. The agent read the wrong haystack. Recall is 1.0, task success is 0.
- Task success passes on a shortcut. The agent answered correctly from prior knowledge without ever calling the retrieval tool. Your "RAG system" is a closed-book model with extra latency and cost.
- Tool-call accuracy passes in isolation. Each call is valid. The sequence is wrong — the agent queries the CRM before resolving the customer ID.
Each of these is invisible to one eval suite and obvious to a joint one.
The 2026 Evaluation Stack: A Five-Layer Architecture
Treat evaluation as five layers that share one task definition and one trace format. Each layer answers a different question, and each one can fail independently.
Layer 1 — Component Metrics (Retrieval + Tool Calls)
Unit-level evaluation of the pieces. For retrieval: context precision, context recall, and rank of the first relevant chunk. For tools: schema validity, argument correctness, and call success rate against a mocked or sandboxed endpoint. These are cheap, fast, and run on every commit. They catch regressions in embedding models, chunkers, rerankers, and tool signatures before anything expensive runs.
Layer 2 — Trajectory Evaluation (Agentic Paths)
Score the path, not just the endpoint. Capture every step as a structured trace — thought, tool, arguments, observation, latency, token cost — then score three things:
- Step-level correctness: was this the right action given the state?
- Path validity: does the sequence match an acceptable plan?
- Path efficiency: how many steps versus the reference plan?
Trajectory scoring is what catches the agent that arrives at the right answer through an unacceptable route — the one that writes to production before reading the record.
Layer 3 — End-to-End Task Success
Outcome-based scoring against task-level ground truth. Binary pass/fail on the final artifact, plus graded partial credit where the task decomposes. This is the number your stakeholders actually care about, and the one you should never let a component metric override.
Layer 4 — Judge Reliability (How You Know the Scores Are Real)
Layers 2 and 3 are usually produced by an LLM judge. If you do not validate the judge, you are measuring the judge's biases, not your system. Before trusting any judged metric, run a calibration protocol:
- Human-labeled anchor set. Hand-label 100–200 traces. Measure judge agreement (Cohen's κ or Krippendorff's α) against humans. Below ~0.6, the judge is not ready to gate releases.
- Bias probes. Test for position bias (swap candidate order), verbosity bias (pad a correct short answer), and self-preference (does the judge favor outputs from its own model family?).
- Judge versioning. Pin the judge model and prompt. A silent judge upgrade invalidates your historical scores.
- Disagreement triage. Every human/judge disagreement is a spec ambiguity. Fix the rubric, not just the score.
Layer 5 — Cost and Latency per Successful Task
A correct answer that costs $0.80 and 40 seconds per call is a failed product. Score it directly:
cost_per_success = total_cost / count(tasks_passed)
latency_p95_success = p95(latency | task_passed)
Report these alongside task success, not in a separate dashboard. Optimizing success without a cost constraint is how teams ship systems that pass eval and lose the deal.
Building the Joint Pipeline
One Task Definition, One Trace Format
Every task in your golden set carries three expectations: expected answer, expected evidence (the specific passages or records that justify it), and expected tool path (the acceptable plan, plus explicitly disallowed sequences). Instrument the runtime to emit one trace per task that all five layers consume. If a layer needs a different data shape, fix the trace schema — do not fork the pipeline.
The Golden Set Is a Product Artifact
Your golden set encodes what "correct" means. Treat it with the rigor of a spec:
- Stratify by difficulty and failure mode, not by convenience. Include the shortcut cases and the empty-context cases on purpose.
- Version it. Every score is only meaningful relative to a golden set version.
- Refresh on drift. When the query distribution or the corpus moves, the golden set goes stale. Schedule review, don't wait for a regression.
What Good Looks Like
A joint eval pipeline earns its keep when a failing task produces a single trace that names the failing layer. Retrieval good, reasoning bad, on a specific step, with the specific tool call that diverged. That is the artifact your team debugs from. Everything above exists to produce it.
The systems that hold up in production are not the ones with the best individual metrics. They are the ones whose evaluation can tell the difference between a lucky pass and a real one — and that requires scoring retrieval and agent behavior against the same ground truth, in the same pipeline, with a judge you have actually validated.