Interviewsretrieval engineerRAG qualityRAG evaluationrecall@k

Inside the Retrieval Engineer Role: How One Team Actually Measures RAG Quality

How one team measures RAG quality with recall@k, a golden query set, and a three-tier scorecard—and why retrieval, not prompts, drives failures.

The Demo Looked Great. Production Didn't.

The demo worked perfectly. Ten curated questions, ten clean answers, a room full of nodding executives. Six weeks later, the same retrieval-augmented generation (RAG) system was fielding real enterprise search queries — messy, acronym-heavy, full of internal jargon — and returning confident answers built on the wrong documents.

The failure mode was almost boring in retrospect. A sales enablement query about "Q3 renewal terms for the enterprise tier" pulled a chunk from a 2022 pricing memo because the phrase "enterprise tier" appeared in both, and the 2022 memo was shorter and therefore ranked higher after chunking. The answer was fluent, cited a real document, and was wrong in the way that costs money. Nobody caught it until a rep repeated expired terms to a customer.

That gap is the reason the retrieval engineer role exists. Not because prompt engineering failed, but because nobody owned the part of the pipeline that decides which context the model sees in the first place.

We sat down with Priya Raman, a retrieval engineer at a mid-size enterprise software company, to walk through how her team actually measures RAG quality — the scorecard they built, the metrics they ignore, and why they stopped treating prompt tweaks as a first-line fix. Her team's system serves internal support, sales enablement, and a customer-facing documentation assistant, running roughly 40,000 queries a day across a corpus of about 900,000 chunks drawn from Confluence, Zendesk, Salesforce knowledge articles, and a public docs site. In their incident reviews, retrieval failures — not generation failures — account for the majority of quality incidents, a split Raman says held steady across the last two quarters of their postmortems. What follows is a practical look at how one team turned "the answers feel off" into a measurable, gated process.

A RAG pipeline diagram with retrieval stages highlighted in a different color from the LLM generatio
A RAG pipeline diagram with retrieval stages highlighted in a different color from the LLM generatio

What a Retrieval Engineer Actually Owns

Raman's scope is narrower than "AI engineer" and deeper than "search tuning." On a given week she touches chunking strategy, embedding model selection and versioning, index configuration, hybrid search weighting, reranking, and query understanding — the rewriting and expansion that happens before a single vector is compared.

Concretely, that means decisions like: whether chunks are 256 tokens or 512, and whether they overlap by 10% or 20%; whether the index is a single vector store or a hybrid of BM25 and dense retrieval with a learned weighting; which embedding model version is pinned, and how a model upgrade gets rolled out without silently shifting every ranking in production. Her team runs a hybrid setup — dense vectors in a managed vector database alongside a keyword index — with a cross-encoder reranker on top of the top 50 candidates, narrowing to the 8 chunks that actually reach the prompt.

The role sits at a seam. ML engineers own model behavior and fine-tuning. Data engineers own ingestion and freshness. Platform teams own latency, uptime, and serving infrastructure. Retrieval engineering owns the contract between the corpus and the model — what gets indexed, how it's represented, and how candidate documents get ranked before they reach the prompt.

That ownership is new. "Two years ago, retrieval was everyone's side project and nobody's job," Raman said. "The pipeline worked in staging, so it never got a dedicated owner. Then the corpus grew, the query distribution shifted, and suddenly retrieval quality was the whole product." Her team made the case for dedicated ownership after tracing a string of bad answers to a single stale index — a failure no individual role was accountable for catching. The index had been rebuilt on a schedule that quietly broke when an ingestion connector changed its document ID scheme, and for eleven days the system served results from a snapshot that no longer matched the source of truth.

What surprised her most about the job:

"I assumed the hard part would be the math. The hard part is the labeling. Getting people to agree on what a correct retrieval even looks like for an ambiguous query is most of the work."

She gives an example: a query like "how do I reset MFA." The literal answer is one doc. But a support agent asking it may want the admin-side procedure, the end-user procedure, and the known-issues page for the bug where resets silently fail. Three labelers produced three different "correct" answers until the team wrote down what the query class was actually for.

Why RAG Quality Is a Retrieval Problem, Not a Prompt Problem

The team's most useful diagnostic habit is a hard split: before anyone touches a prompt, classify the failure. Did the system retrieve the right context and generate a bad answer? Or did it never retrieve the right context at all?

In their incident reviews, the second category dominates. Missing documents, mis-ranked documents, and chunk boundaries that severed the one sentence that mattered — these produce confident, fluent, wrong answers that look like generation problems and aren't. Raman's favorite example: a chunking scheme that split a table of rate limits from the sentence introducing it, so the model dutifully reported numbers with no idea what they applied to. The fix was a chunking rule, not a prompt.

"Prompt tweaking is the default reflex because it's the cheapest thing to try," Raman said. "It's also the thing that hides retrieval bugs. You reword the instruction, the model gets luckier on your test queries, and you've masked a ranking problem that will resurface on the next thousand queries."

The team treats RAG quality as a measurable system property with a defined test set, not a vibe check on a handful of outputs. That framing changes what gets fixed: if the right document isn't in the top k results, no prompt will save you. It also changes who gets pulled into an incident. A bad answer now triggers a retrieval trace review — what was in the candidate window, at what rank, and why — before anyone opens the prompt template.

The Metrics That Team Tracks (and the Ones They Ignore)

Retrieval Evaluation: Offline Before Online

Everything starts with a golden query set — a few hundred real queries sampled from production logs, each paired with labeled relevance judgments. The team maintains it as a living artifact, not a one-time spreadsheet: queries are re-sampled from production logs quarterly to track distribution drift, and every production incident adds a regression case so the set grows toward the failures it has actually seen. The set currently holds about 450 queries, stratified by product area and query class so that a noisy support week can't skew the mix.

Labeling is where the rigor lives. Each query gets a set of acceptable relevant chunks, not a single "correct" document — which matters, because chunk-level and document-level labels produce different numbers from the same system. A retriever can score well at the document level while failing at the chunk level, and chunk-level is what actually reaches the prompt. When two labelers disagree, a third reviewer adjudicates and the team records the disagreement rate as a quality signal on the golden set itself; a rising disagreement rate means the labeling guidelines are underspecified, not that the retriever changed.

On that set they track recall@k (did any acceptable relevant chunk make the candidate window at all?), precision@k, and two ranking metrics used for different purposes: MRR for "how high is the first relevant result" on single-answer queries, and nDCG for queries where several results are relevant to varying degrees. Recall gets the most attention because it's the ceiling on everything downstream: if the answer isn't retrieved, it can't be generated. The team reports recall at two cutoffs — recall@50 for the candidate window before reranking, and recall@8 for what actually reaches the prompt — because a reranker can only recover what the first stage surfaced.

Offline evaluation acts as a ship gate. Any change to chunking, embedding model, index configuration, or reranking runs against the golden set before it goes anywhere near production. A regression in recall@10 blocks the change regardless of how good the demo looked — with one important caveat. Because the golden set is a few hundred queries, small deltas are noise. The gate only fires on regressions that exceed a minimum effect size the team set by measuring run-to-run variance on unchanged code; anything smaller is logged and watched rather than treated as a blocker. In practice that threshold settled at roughly two points of recall@10, which the team derived by re-running the same index five times and observing the spread.

The Three-Tier Scorecard

The team organizes metrics into three tiers, each answering a different question:

  • Tier 1 — Retrieval metrics: recall@k, precision@k, nDCG, and reranker lift. Question: did we find and rank the right context?
  • Tier 2 — Answer-level metrics: groundedness (is every claim supported by retrieved context?), citation accuracy (does the cited source actually say that?), and answer relevance. Question: did the generated answer use the context honestly?
  • Tier 3 — Product metrics: task success, deflection rate, and escalation rate. Question: did the user get what they came for?

Tier 2 deserves a caveat the team is explicit about: groundedness and citation accuracy are scored by an LLM-as-judge, not a human, at production volume. That makes them fast and cheap but not ground truth. The team runs a human-labeled sample against the judge weekly and tracks judge–human agreement; when agreement drops, the judge prompt or model version is treated as a change that needs its own gate. "A judge you haven't validated is just a second model with opinions," Raman said. Their working threshold: if judge–human agreement on the weekly sample falls below about 85%, Tier 2 numbers are flagged as unreliable and excluded from release decisions until the judge is re-validated.

The tiers are diagnostic, not decorative. A drop in Tier 3 with stable Tier 1 points at generation or UX. A drop in Tier 1 explains most Tier 3 drops before anyone opens a prompt file. In one case, a spike in support escalations traced cleanly to a Tier 1 regression on a single query class — queries mentioning a renamed product feature — because the query-understanding layer's synonym list hadn't been updated after a rebrand. Tier 2 was flat, Tier 3 was down, and the scorecard pointed straight at retrieval.

What they deliberately ignore: raw cosine similarity scores as a headline number ("it's a vanity metric — it tells you about the embedding space, not the user's outcome"), and any single aggregate "quality score" that collapses the tiers into one figure. "The moment you average retrieval and answer quality into one number, you've lost the ability to tell which one moved," Raman said. "And the whole point of the scorecard is knowing what to fix."

The result is a process where "the answers feel off" becomes a specific, addressable claim: a Tier 1 regression on a named query class, caught offline, blocked at the gate, and fixed before a user ever saw it.

ShareX / TwitterLinkedIn
← Back to Interviews