Build a Production RAG Pipeline with LangGraph: A Step-by-Step Tutorial

What "Production-Grade" RAG Actually Means in 2026

"Production-grade" is not a model choice. Production-grade RAG is a checklist you either pass or fail. A notebook tutorial that prints one lucky answer proves nothing about the system your users will hit on Monday morning. Here is the checklist we run before any RAG system takes real traffic:

An eval gate blocks deploys below threshold.
A p95 latency budget caps every node.
A cost ceiling bounds every query.
Checkpointed state survives crashes and pauses.
A human approval option guards consequential answers.
Per-node observability traces every decision.

If any item is missing, you have a demo. A demo and a production system differ in failure behavior, not in features.

Three terms carry the weight, so let's define them precisely:

p95 latency — the 95th percentile. p95 latency means 95 percent of requests finish faster than this number. Averages hide your worst user experiences; p95 does not.
Eval gate — a CI step that runs your golden set and fails the build when quality metrics drop below thresholds. The eval gate turns quality into a build status.
Checkpoint — a serialized snapshot of graph state written after each node. Checkpoints make every run resumable and auditable.
Observability — per-node traces capturing inputs, outputs, token counts, and latency. Per-node observability tells you which node broke, not just that something broke.

The anchor table for this entire article:

Requirement	Naive tutorial behavior	Production requirement
Retrieval quality	One hand-picked query works	An eval gate blocks deploys below thresholds.
Latency	"It ran fine on my laptop"	A p95 budget caps each node separately.
Cost	Nobody counts tokens	A cost ceiling bounds every single query.
State	Everything lives in RAM	Checkpointed state survives crashes.
Risk	Auto-executes everything	A human approval option gates risky answers.
Debugging	`print()` statements	Per-node traces capture every decision.

On our last prompt swap, faithfulness moved 12 points on the golden set. Retrieval code changed by exactly zero lines. The eval gate caught the regression before deploy. Without it, we would have shipped a quieter, more confident hallucinator.

Everything below builds toward that checklist, step by step, with code you can run.

Reference Architecture

The system is a state machine, not a chain. A question enters, flows through retrieval and grading, and either reaches generation or loops back through query rewriting. Here is the full topology:

START → retrieve → grade_documents ─┬─ (docs relevant) → generate → evaluate → END
                                    └─ (no docs) → rewrite_query → retrieve

A conditional edge is a function that reads the current state and returns the name of the next node. That single mechanism is what makes corrective loops possible. The graph grades retrieved documents before generating, rewrites the query and retries when grading fails, and falls back to a grounded refusal when the retry budget runs out.

Grading before generation is the highest-ROI decision in the whole graph. Generation is your most expensive node — roughly 1.6 seconds and $0.008 per call at our traffic. Grading costs 180 milliseconds on a small model. A cheap grading call protects an expensive generation call. Skipping the grade means paying full price to hallucinate from irrelevant context.

This design follows two published patterns. Corrective RAG (CRAG) grades retrieved documents and triggers corrective actions on failure. The CRAG paper (Yan et al., 2024) grades each retrieved chunk, then either uses it, re-retrieves with a rewritten query, or falls back to web search. Self-RAG trains the model to critique its own outputs with reflection tokens. Self-RAG (Asai et al., 2023) bakes the critique into the model itself. Our graph takes the pragmatic middle path: CRAG-style grading needs no model retraining. We grade with a small LLM call or an NLI classifier, keep the generator untouched, and get most of the benefit.

Why a State Machine Beats a Linear Chain

Here is the decision rule we apply to every pipeline: If any step can fail and change the path, you need a graph. A linear chain cannot branch on evidence quality. It retrieves, generates, and hopes. When grading fails, a chain has exactly two options — generate anyway or crash. A graph has a third option: rewrite the query and try again, under a budget.

Two more capabilities fall out for free. Cycles enable bounded retry loops without nested code. And interrupts pause the graph for human approval mid-run. You can bolt retries onto a chain with while loops. You cannot cleanly bolt on crash recovery, mid-run approval, and replayable traces. The state machine earns its complexity at exactly these three moments.

The Six Nodes

Each node has a strict input/output contract. Pure functions with typed contracts make every failure reproducible.

Ingest (offline subgraph). Input: raw files. Output: chunk records with deterministic IDs. The ingest node turns raw documents into addressable chunks. It runs on a schedule, not per query, so it lives outside the runtime graph.
Retrieve. Input: question. Output: top-5 documents with scores. The retrieve node turns a question into five candidate chunks. Hybrid search fuses BM25 and dense results via RRF, then a cross-encoder trims 50 candidates down to 5.
Grade documents. Input: question + documents. Output: filtered documents. The grade node filters candidates against the question. A small model or NLI classifier returns a yes/no verdict per document. Empty output triggers the corrective loop.
Rewrite query. Input: question + retry_count. Output: new question, retry_count + 1. The rewrite node transforms a failed query into a better one. It runs at most once per invocation, enforced by the retry budget.
Generate. Input: documents + question. Output: generation. The generate node grounds every answer in retrieved context. If documents are empty after the retry budget, it emits a transparent "I don't know" instead of guessing.
Evaluate. Input: full state. Output: trace metadata. The evaluate node records groundedness signals on every run. Heavy RAGAS scoring runs in CI; this node does a lightweight in-line check and writes to LangSmith.

Prerequisites and Project Setup

Tested with these versions — pin them, because LangGraph's API moved fast across 0.2.x:

Package	Tested version
Python	3.11+
`langgraph`	0.2.60+
`langchain-core`	0.3.x
`qdrant-client`	1.12 (or `pgvector` 0.3)
`ragas`	0.2.x
`langsmith`	0.1.x

You need four credentials:

OPENAI_API_KEY — or any chat + embedding endpoint
QDRANT_URL + QDRANT_API_KEY — or a Postgres connection for pgvector
LANGSMITH_API_KEY — traces and eval runs
DATABASE_URL — Postgres for the checkpointer

Clone and install:

git clone https://github.com/your-org/production-rag-blueprint.git
cd production-rag-blueprint
pip install -r requirements.txt

Version pinning prevents silent API breakage in your graph.

Expert Q&A

Q: How big does the golden set need to be before an eval gate is statistically meaningful? A: Start with 75–150 questions spanning your top query intents, known failure modes, and near-miss phrasings. Below roughly 50 examples, single-digit metric swings are indistinguishable from noise; at 100+, a 10–12 point faithfulness move is a real regression signal. Version the golden set like code, review changes in PRs, and refresh it quarterly — a stale set quietly overfits to your current prompts and stops catching anything.

Q: For document grading, when is an NLI classifier the better choice over a small LLM call? A: Prefer the NLI classifier when relevance is mostly entailment-style and volume is high: it runs in ~10–20 ms, costs effectively nothing, and is fully deterministic. Use a small LLM judge when relevance depends on domain nuance, negation, or multi-hop phrasing — expect 150–300 ms and roughly $0.0001–0.0003 per verdict. A common path is to start with the LLM judge, label a few thousand of its verdicts, then distill it into a classifier once you can measure agreement (Cohen's kappa ≥ 0.7) against the judge on a held-out sample.

Q: Does the Postgres checkpointer add noticeable overhead at p95? A: Typically single-digit milliseconds per node for state snapshots under ~100 KB — invisible next to a 1.6-second generation call. The two things that actually hurt are stuffing full documents into graph state (pass chunk IDs and rehydrate instead) and unbounded checkpoint retention (set a TTL and archive cold traces). For that overhead you get crash recovery, mid-run interrupts, and replayable traces — it is the cheapest reliability you will buy anywhere in the stack.

Q: How do you set the retry budget without blowing the p95 latency budget? A: Work backwards from the number: if p95 must stay under 4 seconds and generation costs ~1.6 seconds, one corrective loop (rewrite + retrieve + re-grade ≈ 0.8–1.0 seconds) fits and two do not. Enforce the cap in the conditional edge itself, not inside the node, and make the grounded refusal the designed terminal state so the worst case is a fast, honest "I don't know" rather than a timeout. Track loop-entry rate as its own metric — if more than ~10–15% of traffic hits the rewrite loop, the defect is in retrieval, not in the budget.

Q: Is RAGAS faithfulness reliable enough to gate deploys on its own? A: No. Faithfulness is claim-entailment scoring: it catches unsupported claims but is blind to retrieval recall and answer completeness, and as an LLM-judge metric it drifts whenever you change the judge model — pin the judge exactly like you pin your packages. Gate on a composite: faithfulness plus answer relevancy and context precision, with a human-labeled holdout of 30–50 items as the tiebreaker. When the metric and the human labels disagree on more than a couple of items, trust the labels and recalibrate the threshold before trusting the gate again.