MLOps & InfrastructureLLM eval harnessprompt regression testingLLM evaluationAI observability

LLM Eval Harness in Production: Building a Regression Suite That Catches Bad Prompts

Build an LLM eval harness that catches bad prompts before they ship. Learn golden datasets, hybrid scoring, thresholds, and CI merge gates.

The silent regression nobody catches

A developer tweaks a system prompt to fix one edge case. The change looks better in the playground, passes a casual eyeball check, and ships. Two weeks later, support tickets spike: the assistant has started dropping required JSON fields on a meaningful share of requests, and a downstream parser has been quietly failing ever since.

This is the defining failure mode of LLM features in production. There's no compiler to catch it, no red squiggle, no stack trace. The output is plausible — which is exactly what makes it dangerous. A dropped field doesn't throw; it returns null, the parser coerces it, and the corrupted record lands in your database where nobody looks until a customer does.

The fix is to treat prompts like code and evals like a test suite. Every prompt change runs against a versioned regression suite before it merges, and pass rates are monitored in production the same way you'd watch error rates. This article walks through building that LLM eval harness: the components, the build sequence, the scoring traps, and the operational discipline that keeps it honest.

Why prompt changes break production silently

The "looks fine in the playground" trap

A playground session tests one prompt against a handful of inputs you thought of in the moment. Production runs that prompt against thousands of inputs you didn't — malformed user text, unusual languages, empty fields, adversarial phrasing, long context. A tweak that improves the case you were staring at can degrade a dozen cases you never sampled. Manual spot-checking measures your imagination, not your coverage.

The arithmetic is unforgiving. If you eyeball ten outputs and a regression affects 5% of traffic, the odds you catch it are about 40% — worse than a coin flip. At 1% incidence, you catch it roughly one time in ten. Meanwhile the change ships, the failure compounds across every request, and the signal you eventually get is a support ticket written in anger, days later, detached from the commit that caused it.

Why unit tests don't translate to non-deterministic outputs

Traditional assertions assume a fixed output. LLMs break that assumption three ways:

  • Non-determinism. The same input yields different outputs across runs, temperatures, and sampling seeds. Even at temperature 0, most providers do not guarantee bit-identical outputs across infrastructure changes.
  • Provider drift. Model versions get updated or deprecated underneath you. A prompt tuned for one snapshot can behave differently on the next. Pin model versions explicitly and treat a version bump as a code change that must clear the gate.
  • Semantic correctness. The right answer can be phrased a hundred ways. assert output == expected is useless; you need scorers that judge meaning, structure, and constraint compliance.

The cost of missing this is cumulative and invisible: degraded answers, rising support volume, eroding trust, and eventually revenue. A prompt change is a production code change — it deserves the same gate. (See: prompt versioning and change management best practices)

Anatomy of a production eval harness

A harness is infrastructure, not a one-off script. It has five parts: a dataset, a runner, scorers, thresholds, and CI integration. Miss any one and it degrades into a notebook nobody runs. The dataset without a runner is a spreadsheet; the runner without thresholds is a report; thresholds without CI are a suggestion. Each part only earns its keep when the others exist.

Golden datasets and regression fixtures

The golden set is a versioned collection of inputs paired with expected properties — not necessarily exact expected strings. Fixtures should include representative traffic, known edge cases, and every historical failure you've fixed. Store them in version control alongside the prompt so a given prompt version always maps to a specific dataset snapshot. (Related: how to version prompts alongside your codebase)

Expected properties, not expected strings, is the key design decision. A fixture for a JSON-extraction task records "output must validate against this schema and contain these field values," not "output must equal this exact string." A fixture for a support reply records "must mention the refund window and must not promise a specific date," not a golden paragraph. Properties survive rephrasing; strings don't.

Two disciplines keep the dataset honest:

  • Prevent leakage. Never copy eval fixtures into few-shot examples, fine-tuning data, or RAG corpora. A model that has seen the test answers will score well and fail in production. This happens most often by accident — someone pastes a good example from the eval set into the system prompt as a demonstration, and every future run on that case is contaminated. Tag fixtures as eval-only and keep them in a separate directory from any data that feeds prompts.
  • Refresh on a cadence. Traffic shifts. Schedule a quarterly review to retire stale cases, add new failure classes, and re-stratify. A golden set that never changes is measuring a product you no longer ship.

Scoring: deterministic checks vs. LLM-as-judge

Use the cheapest reliable scorer for each property:

  • Deterministic checks for anything structural — JSON schema validation, required-field presence, regex constraints, length limits, refusal detection, latency and token budgets.
  • LLM-as-judge for qualities that resist exact matching — tone, factual grounding, instruction adherence, helpfulness.

Hybrid scoring is the default. Deterministic checks catch hard failures cheaply; the judge catches soft regressions. Neither alone is sufficient. (See: LLM-as-judge scoring pitfalls and calibration)

The cost gap is large. A schema check runs in microseconds and costs nothing; a judge call costs tokens and adds latency to every run. Order your scorers cheapest-first and short-circuit: if the JSON doesn't parse, there's no reason to ask a judge whether the tone is right.

Treat the judge as a component under test, not an oracle:

  • Calibrate against humans. Label a sample of outputs by hand and measure judge–human agreement. If agreement is weak, the judge is measuring something other than what you care about. Track agreement as a number — Cohen's kappa in the 0.6–0.8 range is a reasonable target for subjective dimensions — and re-check it whenever you change the judge prompt.
  • Pin the judge model version. A judge upgrade is a measurement change. Re-baseline before you trust the new scores.
  • Use a different model family as judge where practical, to reduce shared blind spots with the model under test.
  • Watch for known biases — verbosity preference, position bias in pairwise comparisons, and self-preference for outputs from the same model.

Baselines, thresholds, and pass/fail gates

Every metric needs a stored baseline per prompt version and a threshold that tolerates normal variance. Measure variance empirically: run the suite N times against a frozen dataset and compute the mean and standard error of each score. Gate on a lower confidence bound — for example, mean − 2 × standard_error — rather than the raw mean. A threshold set at the mean minus one standard deviation will fail roughly one run in six by construction.

Diagram of an eval harness pipeline — golden dataset and prompt version feeding a runner, outputs fl
Diagram of an eval harness pipeline — golden dataset and prompt version feeding a runner, outputs fl

Treat a threshold breach as a failed build — but design for statistical gates. Flaky gates get disabled, and a disabled gate protects nothing. Use a retry policy for borderline runs, log every failure with the trace, and quarantine a test only after a documented investigation, never silently.

Building the regression suite step by step

Step 1 — Instrument production traffic for LLM observability

You can't curate a golden set from data you didn't capture. Log every request and response with the prompt version, model version, latency, token counts, and a trace ID. Capture user feedback signals — thumbs down, regenerations, escalations to a human, downstream parse failures.

The goal is a labeled corpus of real behavior. Observability is the feedstock for evals. (See: LLM observability and tracing setup guide)

Downstream signals are the most valuable and the most neglected. A thumbs-down is a user who bothered to complain; a parser failure, a retry, or a validation error in the consuming service is a failure the user never saw but that still cost you. If your LLM output feeds a database write, a CRM field, or an API call, log the validation outcome of that handoff. That's where the silent regressions from the opening actually surface.

Step 2 — Curate a golden set from real failures

Start with failures, not averages. Pull traces where users gave negative feedback, where downstream systems errored, or where a human reviewer flagged the output. Label the expected behavior — the correct fields, the right refusal, the accurate answer.

Then stratify. A reasonable starting point — adjust to your traffic mix — is roughly:

  • 40% representative happy-path traffic
  • 30% historical failures and edge cases
  • 20% adversarial or malformed inputs
  • 10% known-bad prompts you want the model to refuse or redirect

Keep it small enough to run on every PR. As a rule of thumb, a few hundred well-chosen cases that run on every change beat thousands that run quarterly — but that is a starting heuristic, not a law. Grow the set as you discover new failure classes.

The practical constraint is runtime, not storage. If a run takes 20 minutes, developers stop waiting for it and merge on the red X. Budget for a suite that completes in a few minutes with parallel calls, and if it grows beyond that, split it into a fast subset on every PR and the full set nightly.

Step 3 — Define scorers and acceptance thresholds

For each quality dimension, pick one scorer and one threshold. Be explicit about what "pass" means.

A metric without a threshold is a dashboard. A threshold without a baseline is a guess.

Wire deterministic checks first — they're fast, free, and catch the most embarrassing failures. Then add a calibrated judge for the soft dimensions. Record the baseline score for the current prompt version, and re-baseline deliberately whenever the judge, model, or dataset changes — never as a shortcut to make a failing build pass.

Beware the re-baseline loophole. Every time someone re-baselines to unblock a merge, the gate loses a little credibility. Require a written reason on every re-baseline commit, and review them monthly. If re-baselining is frequent, your thresholds are probably too tight for your variance, and that's a fixable design problem — not a reason to loosen standards.

Step 4 — Build the runner and wire it into CI

The runner executes the dataset against a prompt version, collects outputs, applies scorers, and emits a structured result (per-case pass/fail, aggregate scores, and a diff against baseline). Keep it deterministic in everything except the model call: stable ordering, pinned dependencies, cached fixtures where possible.

Wire it into CI so that:

  • Every pull request touching a prompt, scorer, or model version triggers the suite.
  • Results post as a PR check with a per-case diff, so reviewers see what regressed, not just a red X.
  • The gate blocks merge on threshold breach, with an explicit, logged override path for emergencies.

The per-case diff is the feature that gets the harness adopted. A reviewer who sees "case 47: was passing, now drops the currency field" can judge the change in seconds. A reviewer who sees "score 0.81 < 0.83" learns nothing and starts ignoring the check.

Step 5 — Monitor in production and close the loop

CI catches regressions before merge; production monitoring catches everything CI didn't anticipate — provider drift, traffic shift, and new failure modes. Sample live traffic, run the same scorers asynchro

ShareX / TwitterLinkedIn
← Back to Learn