The LLM Feature Store: Versioning Prompts, Models, and Evals as One Deployable Artifact
Build an LLM feature store that versions prompts, models, and evals as one deployable artifact—atomic promotion, instant rollback, and full audit trails.
Why LLM Apps Break Even When the Model Works
Most LLM applications don't fail because the model is too weak. They fail because the prompts, models, and evals drift out of sync—each one living in a different tool, versioned on a different cadence, owned by a different team. A prompt gets a quick fix in a UI, a model gets swapped in a registry, and the eval suite quietly references a rubric nobody has touched in three months. Everything looks green until a customer hits a regression you can't reproduce.
The fix isn't better documentation or more careful engineers. It's treating prompts, models, and evals as one versioned, deployable artifact—a single unit you can promote, roll back, and audit together. That's the pattern we'll call an LLM feature store. The name borrows deliberately from the classic ML feature store: both are systems of record that guarantee reproducibility across a boundary where things otherwise drift. This article covers why it matters, what it looks like, how to handle the versioning traps that bite real teams, and how to wire it into CI/CD without rebuilding your stack from scratch.
A note on terminology. "LLM feature store" is not yet a standardized product category. What it describes is a composite pattern assembled from existing primitives: a prompt/asset registry (e.g., LangSmith, Langfuse, PromptLayer), a model registry (e.g., MLflow, SageMaker, Vertex), and an eval harness (e.g., OpenAI Evals, Braintrust, DeepEval). Some vendors bundle all three; most teams start by linking the tools they already run. The term is useful because it names the job to be done, not a specific product.
Why LLM Deployments Break Without a Single Source of Truth
The Three Drifting Artifacts
In a typical LLM stack, three things change independently:
- Prompts live in a management UI or a config file, edited by product or prompt engineers.
- Models live in a registry, versioned by ML platform teams on their own release cycle.
- Evals live in notebooks or a separate eval tool, often tied to a dataset snapshot from months ago.
None of these systems share a version. None of them know what the others were tested against. The result is a deployment surface with no reproducible state—you can name the model version in production, but not the exact prompt-plus-eval combination that validated it.
The Cost of Mismatched Versions
The failure modes are predictable and expensive:
- A "prompt fix" ships against a stale model, undoing a safety guardrail the new model had added.
- An eval suite no longer matches production traffic, so passing scores mean nothing.
- A model upgrade introduces silent regressions in edge cases no one re-tested.
- Rollback becomes guesswork—you can revert the model, but which prompt version was it paired with? Which eval proved it was safe?
When you can't reproduce the exact prompt + model + eval combination that shipped, you don't have version control. You have version theater.
The real cost isn't the incident itself. It's the hours spent reconstructing state, the compliance exposure of an unauditable change, and the erosion of trust that makes teams slow down every future release.
What an LLM Feature Store Actually Is
An LLM feature store is a centralized system that stores, versions, and serves prompts, model references, and eval definitions as linked, deployable units. It's the system of record for behavioral artifacts—the things that determine what your LLM application actually does.
Beyond the Classic ML Feature Store
Classic ML feature stores solve training/serving skew: they ensure the features computed in training match those computed at inference. That's a data problem, and the guarantee is feature parity.
An LLM feature store solves a behavior problem. The "features" are prompts, model versions, and eval definitions—the levers that shape model output. Where a classic feature store guarantees feature parity, an LLM feature store guarantees behavioral reproducibility: given a version tag, you can reconstruct exactly what the system did, why, and how it was validated.
The Deployable Artifact: Bundling Prompts, Models, and Evals
The core primitive is a manifest—a small, immutable record that pins:
- Prompt version (content-addressed hash, plus any template variables or tool definitions)
- Model reference (provider, model ID, provider snapshot/version identifier where offered, weights hash for self-hosted models, inference config)
- Eval version (dataset hash, rubric, scorer, and the model used as judge)
- Runtime config (temperature, max tokens, and — for RAG — the retrieval index version)
The manifest is what gets promoted. You don't ship a prompt; you ship a manifest that references a prompt, a model, an eval, and a retrieval index—together. If any component fails its gate, nothing ships.
That last bullet matters more than it looks. For retrieval-augmented apps, the index is a first-class drifting artifact: re-embedding documents or swapping an embedding model changes behavior as surely as a prompt edit does. A manifest that pins the prompt but not the index is only half a reproducibility guarantee.
Versioning Prompts Like Code
Prompt Versioning Semantics and Lineage
Treat prompts with the same discipline as source code. Two mechanisms, used together:
- Content-addressed IDs. Hash the prompt body (and template variables / tool schema). Two teams can't mint the same "1.2.0" for different content, and identical content is provably identical.
- Semantic versioning for humans. Patch for wording tweaks, minor for added variables or tools, major for structural changes that alter the output contract—response schema, tool-call shape, or role definitions. Every major bump requires a fresh eval gate.
Every version is an immutable snapshot with commit-style lineage—who changed it, when, and what it replaced. Attach metadata that matters at deploy time:
- Author and reviewer
- Change rationale (the "why" behind the diff)
- Linked eval results from the run that validated this version
That metadata is what turns a prompt from a string into an auditable artifact.
Immutable Versions, Rollback, and Audit Trails
Immutability is non-negotiable. Once a prompt version ships, it never changes—you create a new version instead. This gives you two things:
- Rollback is a pointer change, not a reconstruction. Revert the manifest and you return to a known-good state—subject to draining in-flight requests and invalidating any prompt/response caches keyed to the old manifest.
- Audit trails satisfy compliance and incident response. Every production change traces to a version, an author, and a passing eval run.
Immutable prompt versions turn rollback from a forensic exercise into a pointer swap.
Tying the Model Registry to Prompt and Eval Versions
Model Registry as the Deployment Anchor
The model registry remains the deployment anchor—it's where model versions are published, tagged, and served. But in an LLM feature store, every model version carries provenance the manifest can reference:
- For self-hosted models, a weights hash and serving config.
- For hosted/API models, a provider model ID plus the provider's snapshot or version identifier. This is the reproducibility trap most teams miss: providers can change behavior under the same model ID. Pin the snapshot where the provider offers one, and treat "same ID, new date" as a drift event that triggers re-eval rather than a silent upgrade.
The registry and the feature store stay loosely coupled: the registry owns model lifecycle and serving; the feature store owns the binding between a model version and the prompts and evals it was validated against. That binding is what makes a deployment reproducible.
Compatibility Pinning
Not every prompt works with every model. Record explicit compatibility constraints in the manifest—for example, "prompt p-9f2 requires model family claude-sonnet-4 or later; tool schema v3." A model upgrade that violates a constraint fails the gate before it reaches production, instead of failing a customer conversation.
Versioning Evals as First-Class Artifacts
Evals are where most "versioned stacks" quietly fall apart. A passing eval is only meaningful if you can reproduce what was tested and how it was scored. That means pinning four things, not one:
Dataset Versioning and Contamination Control
Version eval datasets with the same content-addressing as prompts. More importantly, control contamination: if the same examples used to iterate on a prompt are also used to certify it, your scores are inflated. Keep a held-out, versioned "certification" set that prompt authors cannot see, and run it only at the gate. Rotate it deliberately and track its lineage.
Rubric and Scorer Versioning
- Deterministic scorers (exact match, schema validation, regex) version trivially—hash the code.
- LLM-as-judge scorers do not. The judge is itself a model whose behavior can change. Pin the judge model snapshot, the judge prompt, and the scoring rubric as part of the eval version. A rubric edit is an eval version bump, full stop.
Statistical Reality: Flake and Significance
LLM outputs are stochastic. A single run's score is not a verdict. Gate on a sample size with a confidence interval, and set thresholds in terms of the lower bound, not the point estimate. Track per-example flake rates so you can distinguish a real regression from noise—otherwise teams learn to re-run until green, which defeats the gate entirely.
Gating Thresholds
Define per-eval thresholds in the manifest's gate config: minimum pass rate, maximum regression on any tracked slice, and hard-fail categories (e.g., safety or schema violations) that block regardless of aggregate score. Slicing matters—an aggregate that holds while one customer segment craters is a silent regression.
Wiring It Into CI/CD
The manifest is the unit of deployment, so it's also the unit of the pipeline:
- Author changes a prompt or eval → creates an immutable, content-addressed version.
- Gate runs the pinned eval suite (held-out certification set, pinned judge, statistical thresholds) against the candidate manifest.
- Promote on pass: the manifest moves through environments (dev → staging → prod) as a pointer, never a rebuild.
- Observe in production: log the manifest ID with every request so any incident traces back to an exact, reproducible state.
- Roll back by reverting the pointer to the last passing manifest—prompt, model, and eval revert together.
The payoff is that a production incident becomes a lookup, not an investigation: you have the manifest ID, so you have the prompt, the model snapshot, the retrieval index version, and the eval that certified it. Reproducibility stops being a promise and becomes a property of the system.
Getting Started
You don't need to rebuild your stack. Start by making the three artifacts referenceable: content-address your prompts, pin model snapshots (including hosted-model version identifiers), and version your eval datasets, rubrics, and judges. Then introduce the manifest as the thing you promote, and gate it in CI. The tools can stay where they are—the feature store is the binding layer that makes them one artifact.