Subscribe
TutorialsRAG agentretrieval-augmented generationhybrid searchreranking

Build a Production-Ready RAG Agent in 2026: A Step-by-Step Tutorial

Build a production-ready RAG agent in 2026: retrieval design, hybrid search, reranking, evals, security, and deployment — step by step.

Algorithmine Team
··Updated Oct 26, 2026·6 min read·85 views

filters = {"access_level": {"$in": user_acl}} if user_acl else {}

dense_hits = vector_store.search(embed(query), top_k=k_candidates, filter=filters)
sparse_hits = keyword_index.search(query, top_k=k_candidates, filter=filters)

fused = reciprocal_rank_fusion(dense_hits, sparse_hits, k=60)
reranked = reranker.rank(query, fused[:k_candidates])
return reranked[:k_final]


### The Retrieval Metrics You Must Track

Instrument retrieval before you touch generation:

- **Hit rate @ k:** Did the correct chunk appear in the top *k*? This is your ceiling — if it's not retrieved, it can't be cited.
- **MRR / nDCG:** How highly was the correct chunk ranked?
- **Context precision and recall:** Of the chunks passed to the generator, how many were relevant, and did you capture all the relevant ones?

Track these per query type (lookup, multi-hop, comparative). A single aggregate number hides the failure modes that matter.

## Step 2 — Build the Agent Orchestration Loop

The orchestration layer decides when to retrieve, when to call a tool, and when to answer. The most common production bug here is an **unbounded loop** — the agent keeps calling tools until it exhausts your budget.

### Termination Conditions Are Non-Negotiable

Every agent loop needs hard stops:

```python
MAX_STEPS = 6
MAX_TOKENS = 12_000
MAX_WALL_CLOCK_S = 20

def run_agent(query, budget):
    state = {"steps": 0, "tokens": 0, "start": now()}
    while True:
        if state["steps"] >= MAX_STEPS:
            return fallback_answer(query, reason="step_limit")
        if state["tokens"] >= MAX_TOKENS or elapsed(state) > MAX_WALL_CLOCK_S:
            return fallback_answer(query, reason="budget_limit")

        action = llm.plan(query, history=state["history"], tools=TOOL_SCHEMAS)
        if action.type == "answer":
            return action.content
        result = execute_tool(action, user_context=state["user"])
        state["history"].append((action, result))
        state["steps"] += 1

The fallback_answer path is what separates production from prototype. When the agent can't resolve a query within budget, it should say so — not loop until the request times out.

Tool Design

Expose a small, well-typed tool surface. Each tool needs a clear description, typed parameters, and a defined failure mode. A retrieval tool, a document-fetch tool, and a calculator cover most enterprise use cases. Resist the urge to expose twenty tools; every added tool increases the chance the model picks the wrong one.

Caching

Two caches pay for themselves immediately:

  • Embedding cache: Identical or near-identical queries recur constantly in production. Cache embeddings keyed by normalized query text.
  • Answer cache: For read-heavy workloads, cache full responses keyed by query + user permission scope. Invalidate on document updates.

Step 3 — Generation with Grounding and Citations

The generator's job is narrow: answer using only the retrieved context, cite sources, and refuse when the context is insufficient.

Prompt Contract

Enforce a strict contract:

  • Answer only from provided context.
  • Attach a citation to every factual claim.
  • If context is insufficient, say "I don't have enough information" — do not fall back on parametric knowledge.
  • Keep the answer concise; long answers invite unsupported elaboration.

Structured Output

Require the model to return structured output — answer text plus a list of cited chunk IDs — so you can validate citations programmatically. A citation that points to a chunk not in the retrieved set is a hallucination and should trigger a retry or a refusal.

Faithfulness Scoring

Score every response for faithfulness (does the answer follow from the cited context?) using a cheap judge model with a rubric. Log the score. A drop in mean faithfulness after a prompt or model change is your earliest regression signal.

Step 4 — Evaluation: Offline and Online

Evaluation is the harness that turns a demo into a system. Build it before you need it.

Build a Versioned Test Set

Curate 100–300 real questions with known-correct answers and source documents. Include adversarial cases: out-of-scope questions, ambiguous phrasing, multi-hop queries, and questions whose answers changed recently. Version this set like code — every eval run references a specific version.

LLM-as-Judge, With Guardrails

LLM-as-judge is fast and cheap, but it has known biases: position bias (favoring the first option), verbosity bias (favoring longer answers), and self-preference (favoring outputs from the same model family). Mitigate by:

  • Using a rubric with explicit criteria rather than a holistic score.
  • Randomizing option order in pairwise comparisons.
  • Spot-checking a random sample with human reviewers and tracking judge-human agreement. If agreement drops below your threshold, recalibrate the rubric.

Online Evaluation

Offline evals catch regressions; online signals catch reality. Track:

  • Thumbs up/down and explicit corrections from users.
  • Refusal rate — a spike means retrieval is failing or the prompt is too conservative.
  • Escalation rate — how often users abandon the agent for a human.
  • Faithfulness score distribution on live traffic (sampled).

Step 5 — Observability and Cost Control

You cannot debug what you cannot see. Every request should produce a trace containing the query, retrieved chunk IDs and scores, tool calls, the final prompt, token counts, latency per stage, and the response.

The Cost Formula

Know this decomposition cold:

cost_per_query =
    embedding_cost
  + vector_search_cost
  + rerank_cost
  + generation_tokens × generation_price
  + judge_tokens × judge_price   # if scoring online

Work a concrete example: a 1,500-token prompt and 300-token answer on a mid-tier model at $3/M input and $15/M output is roughly $0.0045 + $0.0045 ≈ $0.009 per query before retrieval and reranking. At 50,000 queries/day, that's ~$450/day in generation alone — which is why caching and a cheaper model for simple queries are not optional at scale.

Budgets and Alerts

Set per-request token caps (enforced in code, not just the prompt), per-user rate limits, and daily spend alerts. A single runaway loop can burn through a monthly budget in hours if nothing stops it.

Step 6 — Security, Permissions, and Multi-Tenancy

Security is where RAG systems fail most expensively. A retrieval that ignores permissions is a data breach with extra steps.

Enforce Permissions at Retrieval Time

Filter before search, using metadata the ingestion pipeline attached. Three patterns, in order of isolation strength:

  1. Separate indexes per tenant — strongest isolation, highest operational overhead.
  2. Namespaces within a shared index — good isolation, moderate overhead.
  3. Metadata filtering on a shared index — lowest overhead, requires rigorous filter enforcement in every code path.

Whichever you choose, the filter must be applied in the retrieval function itself — not passed as an optional argument that a caller can forget.

Other Non-Negotiables

  • Prompt injection defense: Treat retrieved content as untrusted. Strip or escape instructions embedded in documents, and never let retrieved text override system instructions.
  • PII handling: Detect and redact PII at ingestion and in logs.
  • Audit logging: Record who retrieved what, when, for every request. This is both a compliance requirement and your best debugging tool.

Step 7 — Deployment and the Feedback Loop

Ship Behind a Gate

Deploy with a canary: route 5% of traffic to the new version, compare faithfulness, refusal rate, latency, and cost against the baseline, then expand. Never ship a prompt or model change without an eval run attached to the deploy.

Close the Loop

The feedback loop is what keeps the system accurate after launch:

  1. Capture user corrections and low-faithfulness responses.
  2. Triage failures into retrieval, prompt, model, or data categories.
  3. Fix the root cause — often it's a chunking or metadata problem, not the model.
  4. Add the failure case to the versioned test set.
  5. Re-run evals and redeploy.

Teams that run this loop weekly compound their accuracy. Teams that skip it watch their agent quietly degrade as the corpus drifts.

The Bottom Line

A production RAG agent is an engineering system, not a prompt.

ShareX / TwitterLinkedIn
← Back to Learn