In the Trenches: How One Team Cut Agent Hallucinations by 78% in Production

If you run AI agents in production, you already know the feeling. The output looks right. It reads fluently, cites the right customer names, references the right SKUs. And then someone downstream discovers the dollar figure was off by a factor of ten — or that the "approved" compliance step never actually happened.

That is the "confident wrong" problem, and it nearly sank Elena Moreau's deployment.

Elena is the Lead ML Engineer at a fintech platform that handles customer operations, reconciliation workflows, and support triage. When we talked, she was candid about how bad it got before it got better — and exactly how her team clawed production reliability back from the hallucination problem.

"We stopped blaming the model and started measuring the pipeline"

Elena's story begins where many do: an agent that worked in demos and failed in production. Her team's multi-step agent handled routine account operations and support actions. In testing, it was impressive. Live, it was dangerous.

"Around one in five outputs was hallucinated on our agentic workflows," she told us. "And the worst part wasn't the frequency. It was the confidence. The model never sounded unsure. It would confidently quote a balance, confidently reference a policy that didn't exist, confidently describe a completed action that never ran."

Industry numbers back her up. Across enterprise deployments, average hallucination rates hover around 20%, and multi-step agent workflows can spike to 20–40%. For Elena's team in a regulated financial context, each serious incident carried a remediation cost that reached five figures.

The breakthrough wasn't a new model. It wasn't a bigger prompt. It was a decision to treat hallucination as an engineering discipline rather than a model-quirk.

"The first thing we did was stop apologizing for the model and start measuring our pipeline," Elena said. "You can't fix what you haven't instrumented."

Pillar 1: Grounding is a contract, not a suggestion

The single highest-leverage fix, Elena says, was changing how the agent was grounded — and how it was allowed to answer.

"We stopped treating retrieved documents as vibes and started treating them as a contract," she explained. "The system prompt now says, in effect: your answer must be attributable to the provided sources. If you cannot attribute it, you do not invent it. Full stop."

That shift is backed by a well-established finding: grounding the model in retrieved passages with a strict citation contract is the most reliable single strategy for reducing hallucination. But Elena emphasizes that grounding only works if the retrieval layer is trustworthy, too. Her team rebuilt it in three concrete ways:

Hybrid search. They combined dense embeddings for semantic similarity with lexical search for exact proper-noun and ID matching. "Entity names and account numbers were getting mangled by pure semantic search. Hybrid fixed that overnight."
Task-shaped chunking. Instead of fixed-size chunks, they split documents into semantically coherent units — sections, bullet groups, tables — with overlap. "A self-contained chunk means the model doesn't have to stitch two unrelated fragments into a guess."
Reranking. They added a reranker to re-evaluate the top-k retrieved chunks. "Reranking alone chopped another ~20% off our residual errors. The right passage was usually in the top 10; the reranker got it into the top 3."

Pillar 2: Retrieval evals — the guardrail against ourselves

Once grounding improved, Elena's team did something most teams skip: they built an evaluation harness and made it the gatekeeper for every change.

"We treat the eval harness as our CI for reliability," she said. "Every prompt tweak, every chunking change, every new tool — it has to pass the groundedness regression suite before it touches production."

Their eval suite tracks a groundedness metric: a score for whether each claim in an output is supported by a retrieved source. They built a labeled test set of tricky real-world cases — ambiguous policy questions, account-number lookups, edge cases where the model historically hallucinated.

"Once we could measure groundedness on every release, we stopped having arguments about whether something was 'better.' We had a number."

Pillar 3: Guardrails and the power of "I don't know"

The most counterintuitive and highest-leverage change, Elena says, was teaching the agent to refuse.

"We taught the agent to say 'I don't know,'" she said. "It was our single most impactful change. The confident-wrong category basically disappeared, because we removed the confident part."

Concretely, her team layered several guardrails:

Abstention scaffolds. If confidence in a factual answer is too low — or the answer isn't supported by retrieved sources — the agent states uncertainty or refuses rather than guessing.
Entity validation. Before the agent quotes a balance, references a policy, or names a customer, a validation gate cross-checks the entity against internal databases. "If the SKU doesn't exist in our catalog, the agent hears about it before the user does."
Confidence routing. Low-confidence outputs don't reach the user directly. They're routed to a more cautious path, or to a human reviewer.
Runtime guardrails. Business rules enforced at the framework level, so the model literally cannot bypass a non-negotiable constraint. "It's not persuasive; it's hard-coded at the access layer."

Pillar 4: Multi-agent validation and knowing when a human must look

For the highest-stakes outputs — anything touching money movement or compliance — Elena's team added redundancy.

"We use a second pass for critical actions," she said. "A second agent, or a verification step, checks the first agent's answer against the sources. Two independent passes catch the single-pass error that one model would have shipped."

And for the truly consequential decisions, they kept humans in the loop.

"Abstention, guardrails, and validation get us from one-in-five to one-in-twenty," Elena said. "But when a wrong answer costs six figures in a regulated context, a human review for that specific class of action is not a failure of automation. It's the price of trust."

The numbers: 78% fewer hallucinations in production

Elena's team tracked their progress over a 90-day window against their baseline.

"At baseline, our hallucination rate on agentic workflows was roughly 18–20%," she said. "After the four pillars were in and stabilized, we were at about 4–5%. That's a 78% relative reduction. Crucially, we got there without a single rollback — the eval harness caught every regression before it shipped."

The operational payoff compounded:

Trust recovered. Support and ops teams stopped double-checking every agent action.
Incident burden dropped. The five-figure remediation incidents largely disappeared.
Deflection improved. With confident-wrong outputs gone, the agent could safely handle more routine work end-to-end.

Do this first: a starter playbook

If you're fighting hallucinations today, Elena's team suggests this order — because each pillar makes the next one easier to measure:

Instrument and baseline. Measure groundedness and hallucination rate before you change anything. Pick a labeled eval set of your worst real cases.
Harden grounding. Hybrid search, task-shaped chunking, a strict citation contract, and a reranker.
Add a retrieval eval gate. Make groundedness passes a requirement for every change.
Layer guardrails and abstention. Entity validation, confidence routing, runtime rules — and permission to say "I don't know."
Add second-pass validation for high-stakes actions. Human review where the cost of being wrong is highest.

What we'd do differently

Asked for hindsight, Elena offers one honest regret.

"We shipped the eval harness later than we should have," she said. "We spent the first month persuading ourselves things were 'probably fine.' A groundedness eval from day one would have saved us weeks and a couple of incidents. Measure first. Always."

And her closing advice for teams still stuck:

"Stop asking 'how do I stop the model from hallucinating?' and start asking 'how do I make my pipeline incapable of shipping a confident wrong answer?' The model is one layer. The system around it is where you win."

What numbers are you seeing in production? Benchmark against the 78% reference in the comments — or share how you're measuring groundedness on your own agent pipelines.

Interview conducted by the Algorithmine editorial team. This article is part of our "In the Trenches" interview series on real-world AI engineering.