RAG Evaluation Beyond Recall: Metrics That Predict Whether Users Trust the Answer
Recall@k hides trust failures. Learn the RAG evaluation metrics—groundedness, hallucination rate, calibration—that predict whether users trust answers.
Why a Recall@k Win Can Still Mean a Trust Loss
Teams celebrate a recall@k improvement in the dashboard, ship the change, and then watch users quietly stop trusting the answers. Support agents start double-checking every response. Analysts copy the citation, then ignore the claim. Nobody files a bug — they just route around the system. That is a trust failure, and retrieval metrics never saw it coming.
The reason is structural. Retrieval metrics measure whether the right chunk was found; trust depends on whether the answer is grounded, complete, and honest about what it doesn't know. A retriever can hit 0.95 recall and the generator can still overclaim, drop a critical caveat, or cite a passage that doesn't support the sentence it's attached to.
This article lays out the RAG evaluation metrics that actually predict user trust, how to build an offline harness around them, and how to connect those scores to online behavior so regressions get caught before users do. (See also: How to Build a RAG Evaluation Harness)
Why Recall Alone Fails as a Trust Proxy
The Retrieval-to-Answer Gap
Recall@k answers one question: did the relevant chunk appear in the top-k? It says nothing about whether the model read it, prioritized it over a stale or contradictory passage, or reproduced its content faithfully. A high-recall pipeline with a fluent generator can still hallucinate a number, invert a policy, or assert a conclusion the source explicitly qualifies.
The gap widens as answer complexity grows. Multi-hop questions, questions with an implicit "it depends," and questions whose correct answer is "the documentation is ambiguous here" all pass retrieval checks and fail users. Recall is a necessary input to trust, never a sufficient signal for it.
What "Trust" Actually Means to a User
Users rarely evaluate an answer against ground truth — they don't have it. They evaluate whether it feels safe to act on. In practice, that composite has five components:
- Correct — the answer matches reality, not just the retrieved text.
- Grounded — every claim traces to evidence the user can inspect.
- Complete — the sub-parts of the question are actually addressed.
- Appropriately hedged — uncertainty is stated where it exists, not smoothed over.
- Traceable — citations resolve to the passage that supports the specific claim.
Treat trust as a measurable composite, not a vibe. Each component maps to a metric you can compute offline and approximate online. When one degrades, users notice — usually before your dashboards do.
A user who has to verify one answer starts verifying all of them. The cost of a single overconfident hallucination is rarely one bad answer; it's the permanent tax of distrust on every answer after it.
The Trust-Predictive Metric Stack
Faithfulness and Groundedness
These two are related but distinct, and conflating them hides real failures:
- Faithfulness — is each claim supported by the retrieved context as a whole? This catches the generator inventing content the sources never contained.
- Groundedness — is each claim supported by the specific passage it cites? This catches the subtler failure where the answer is technically supported somewhere in the context, but the citation attached to it points at the wrong place.
Score both at the claim level, not the answer level. Decompose the answer into atomic claims, then run entailment checks of each claim against (a) the full context for faithfulness and (b) the cited passage for groundedness. Answer-level pass/fail hides the pattern that matters: an answer that is 90% grounded usually contains the one unsupported sentence a user will act on. Claim-level scoring also localizes failures, so you can tell whether the retriever missed evidence or the generator ignored it. (Related: Claim-Level Groundedness Scoring Explained)
Judge reliability is the load-bearing assumption here. Off-the-shelf NLI models and LLM-as-judge setups both carry meaningful error rates on domain text, and LLM judges show position and verbosity bias. Before you trust these scores, calibrate the judge against a few hundred human-labeled claim/evidence pairs, report agreement (e.g., Cohen's κ), and pin the judge model and prompt version. A faithfulness score you can't reproduce is worse than no score.
Hallucination Rate (and Its Cousins)
Split hallucination into two tracked rates. Intrinsic hallucination contradicts the retrieved context; extrinsic hallucination adds content the context never mentions. They tend to have different fixes — intrinsic errors are often prompting or decoding problems (or genuine context conflicts), while extrinsic errors are often retrieval-coverage problems or parametric-knowledge leakage. Treat that mapping as a starting hypothesis for diagnosis, not a root-cause verdict.
Track the rate per query class, not as a single global number. A 2% overall rate can hide a 30% rate on multi-hop questions, which is exactly where users are most likely to trust the answer blindly. Pair the rate with a false-confidence metric: the share of hallucinated answers delivered without hedging. That combination — wrong and confident — is what converts a bad answer into a lost user.
Answer Relevance and Completeness
Relevance asks whether the answer addresses the question asked. Completeness asks whether the required sub-parts are covered. Together they catch the "technically true but useless" answer that recall never flags: a correct definition when the user needed the exception, or a policy summary that omits the eligibility condition.
Operationalize completeness by decomposing the golden-set question into required facets and checking coverage. For a question like "Can I expense this international flight?", the facets are eligibility, approval path, and documentation — an answer covering two of three is a completeness failure even if every sentence is grounded.
Context Utilization and Citation Accuracy
Two cheap diagnostics close the loop between retrieval and generation. Context utilization measures whether the retrieved evidence actually influenced the answer — low utilization with high recall means you are paying latency and token cost for chunks the model ignores. Citation accuracy measures whether inline citations resolve to the passage that supports the specific claim they're attached to.
Citation accuracy is the metric users feel most directly, and it has two distinct failure modes worth tracking separately:
- Misattribution — the citation resolves to a real but irrelevant passage. This is arguably worse than no citation, because it launders an unsupported claim behind the appearance of evidence.
- Omission — a claim that needed a citation has none. Users learn to distrust uncited sentences, so silent omission erodes trust almost as fast as misattribution.
Score citation accuracy at the claim-citation pair level: for each cited claim, does the cited passage actually entail it? Report misattribution and omission rates separately so you can tell whether the generator is citing wrongly or forgetting to cite.
Building the Offline Harness
A metric only predicts trust if it is stable, calibrated, and wired to outcomes. The harness has four parts:
- A stratified golden set. Do not sample randomly. Build query classes that reflect real traffic (multi-hop, policy-exception, ambiguous-documentation, out-of-scope) and weight them by production frequency. A golden set that is 90% simple lookups will report a healthy score while the hard classes rot.
- Claim-level scoring with a pinned judge. Fix the judge model, prompt, and decoding settings. Re-run a calibration subset against human labels on every judge upgrade; if agreement drops, the score is not comparable across releases.
- Per-class thresholds, not global ones. A single pass/fail line hides the class that matters. Set thresholds per query class from the calibration data — e.g., faithfulness ≥ 0.95 on simple lookups but a stricter false-confidence ceiling on multi-hop.
- A regression gate tied to the metric stack. Block deploys on faithfulness or false-confidence regressions in any class, not just on average recall. Recall can improve while faithfulness drops; the gate must be able to see that.
Connecting Offline Scores to Online Trust
Offline metrics only earn the word "predictive" once you show they track real behavior. Instrument the online proxies that map to the five trust components:
- Edit and override rate — how often users rewrite or discard the answer. The closest online proxy for groundedness and completeness failures.
- Citation click-through and dwell — users clicking a citation and bouncing fast are verifying, not trusting.
- Escalation and re-ask rate — a re-asked question within the same session is a trust failure in real time.
- Thumbs-down with reason tags — route to the offline query class so a spike is attributable to a metric, not just a mood.
Then run the correlation you actually care about: does a drop in offline faithfulness or a rise in false-confidence precede a rise in edit/override rate? If it does, you have a predictive metric. If it doesn't, you have a vanity metric wearing a lab coat. Recalibrate thresholds quarterly — trust is a moving target as content, models, and user expectations drift.
The Takeaway
Recall tells you whether the evidence was retrieved. It does not tell you whether the answer is worth trusting. The metrics that predict trust — claim-level faithfulness and groundedness, per-class hallucination and false-confidence rates, facet completeness, and claim-level citation accuracy — measure the failure modes users actually feel. Build the harness around a stratified golden set and a calibrated judge, gate deploys on per-class regressions, and prove the metrics predict online behavior before you call them predictive. A recall@k win that ships a trust loss is not a win.