From 5 Evals to 5,000: Inside the QA Team Making Enterprise LLMs Trustworthy at Scale
The author is the QA Platform Lead on the Platform Engineering team, with 15 years in software testing and 6 years building LLM evaluation systems. All figures in this article come from internal run l
By the QA Platform Lead, Platform Engineering · 15 years in software testing, 6 years building LLM evaluation systems · All figures come from our internal run logs and cost reports.
People-first summary: If your enterprise relies on large language models, you need a system that tests them continuously. This article shows how one team built that system — the hiring, the architecture, and the cost math — so you can adapt it to your own stack.
The Spreadsheet That Started It All
Every team has an origin story. Ours began with a single engineer, a spreadsheet, and a checklist that lived on a shared drive.
That engineer ran exactly five evaluations. Each one was hand-written. Each one required a manual run. Each one produced results that nobody fully trusted. When a prompt changed, the checklist had to be updated by hand. When a model was swapped, the whole ritual started over.
It worked. For about three months.
Then the product team shipped a new feature. The checklist broke. A regression slipped through silently. Customers noticed before we did.
That was the inflection point.
Scaling from 5 evals to 5,000 was never really about the evals. It was about building an organization that could trust its models at enterprise scale. It was about turning quality assurance from a manual chore into an engineering discipline.
Here is the thesis: trust at enterprise scale is not a model property. It is an engineering discipline. The model is only as trustworthy as the system that tests it.
This is the story of how one QA team built that system — the architecture, the hiring, the cost math, and the hard lessons along the way.
Meet the Team: How the QA Org Took Shape
From One Engineer to a Cross-Functional Pod
The solo-engineer model died fast. One person cannot maintain 5,000 evals. One person cannot judge ambiguous outputs. One person cannot own infrastructure, safety, and product judgment simultaneously.
The first step was embedding QA into product pods. Each pod got a dedicated QA engineer. That engineer worked alongside the ML engineers and prompt engineers building the feature.
The second step was forming a central LLM QA function. This team owned the eval harness, the golden datasets, and the shared infrastructure. Product pods owned their domain-specific evals. The central team owned the platform.
This split worked because it matched incentives. Product pods cared about their feature. The central team cared about the whole system. Both had clear ownership. Neither could blame the other.
The reporting structure mattered too. QA did not report to product. QA did not report to data science. QA reported to platform engineering. That kept the team independent from the people shipping features.
The Hiring Profile: What an "AI QA Engineer" Actually Looks Like
"AI QA engineer" sounds like a buzzword. In practice, it is a precise blend of three skill sets.
First, software testing rigor. These engineers understand test design, assertion-based checks, and regression discipline. They treat evals like unit tests because evals are unit tests.
Second, ML fluency. They understand model behavior, embeddings, sampling, and scoring. They know why an LLM-as-judge can be biased. They know when it cannot be trusted.
Third, product judgment. They can read an ambiguous output and decide whether it is acceptable. They can translate a product requirement into a measurable eval.
The hiring bar was high. We looked for people who had broken production systems and learned from it. We looked for people who were skeptical of their own tests. We looked for people who could explain a flaky eval without blaming the model.
Domain experts rounded out the team. A healthcare customer meant a clinician in the pod. A finance customer meant an actuary. These experts did not write code. They wrote golden cases and judged edge cases.
The Architecture: From Manual Checklist to Eval Pipeline
The spreadsheet had to die. It was replaced by an eval harness that runs thousands of cases on every change.
The Eval Harness: Golden Datasets, Unit Tests, and Live Evals
The eval harness is the engine. It executes thousands of test cases and scores the results. It is the workhorse of the entire system.
Three kinds of evals feed the harness.
Golden datasets are curated, human-validated cases. Every one has a known-good answer. They are the source of truth for regression testing. If a change breaks a golden case, you know immediately.
Unit tests are assertion-based evals. They are deterministic. They check format, JSON validity, tool-call correctness, and required fields. These are the easiest to write and the fastest to run. They catch the mechanical failures.
Live evals sample real production traffic. These are scored with an LLM-as-judge, then spot-checked by humans. They catch the fuzzy failures that unit tests miss.
The balance matters. Golden datasets anchor the system. Unit tests keep it fast. Live evals keep it honest.
CI/CD Integration: Evals as Deployment Gates
Evals are useless if nobody runs them. So we wired them into the pipeline.
Every pull request triggers the eval suite. The harness runs the full set of relevant evals. A pass-rate threshold decides the outcome.
If evals pass, the change merges. If evals fail, the merge is blocked. The developer sees the failing cases and the reasoning. No model ships without passing its evals.
This is the critical shift. Evals are not a review step at the end. They are a gate at every step. They are a deployment gate, exactly like a test suite for traditional software.
Regression Testing and Drift Detection
Models change. Prompts change. Data changes. All of it can silently break behavior.
Regression testing catches these silent regressions. When a prompt is edited, the harness runs the full golden suite. A drop in pass rate flags the change. A fine-tune or model swap triggers the same check.
Drift detection watches the long arc. The harness monitors eval pass rate over time. A slow decay triggers an alert. The team investigates before the decay becomes a customer incident.
The alert is the key difference between reacting and preventing.
The Cost Math: What 5,000 Evals Actually Cost
Nobody talks about the cost. Let's fix that.
Every eval is a set of model calls. Every model call costs money. Multiply that by thousands of cases, run on every PR, and the bill climbs fast.
We track three cost dimensions.
Compute cost is the raw spend on model inference. Every eval run is tokens in and tokens out. The bigger the model, the higher the cost per eval.
Latency cost is the time cost. Thousands of evals take minutes to run. That time sits in the developer's deploy path. Slow evals slow down every engineer.
Maintenance cost is the human cost. Evals rot. Golden cases go stale. Thresholds drift. Someone has to keep the suite healthy, or it becomes noise.
The trick is not to eliminate cost. It is to spend deliberately.
We tiered the evals. Fast, cheap unit tests run on every PR. Expensive golden and live evals run on merge and on nightly schedules. We cached evals that did not change. We sampled live traffic instead of scoring everything.
The result: a 10x increase in eval count, but only a 3x increase in cost. Discipline beats volume.
The Hard Lessons: What Failed Along the Way
Scaling evals is not a straight line. We made mistakes. Here are the ones worth stealing.
Lesson 1: Evals Are a Trust Problem, Not a Test Problem
The first version of the harness was technically correct and organizationally useless. Nobody trusted the numbers.
The problem was not the code. It was the lack of a shared definition of "good." Engineers, product, and QA each had a different mental model of what the model should do.
We fixed it by writing a shared rubric. One document. One definition of acceptable output. Every eval traced back to it. Trust came from alignment, not automation.
Lesson 2: The LLM-as-Judge Is Not a Judge
We leaned hard on LLM-as-judge scoring. It is fast, cheap, and scales. Then it lied to us.
A judge model has its own biases. It rewards verbosity. It is lenient on some error classes and harsh on others. It drifts when the underlying model changes.
We stopped treating it as ground truth. We made it a filter, not a verdict. Every judge score gets human spot-checks. Anything near a threshold goes to human review. The judge narrows the problem; the human owns the decision.
Lesson 3: Thresholds Are a Trap
We set a pass-rate threshold and celebrated. Then the team learned to game it.
Developers tuned prompts until the eval passed. They did not make the model better; they made the test easier. The threshold became a target, and the target became a lie.
We responded with adversarial evals. We added cases designed to break the model. We rotated the golden set so the suite could not be memorized. And we watched the trend, not the single number.
Lesson 4: Flaky Evals Destroy Trust Faster Than Bugs
A flaky eval fails sometimes and passes other times with no code change. Nothing erodes confidence faster.
When an eval flakes, developers ignore it. When enough evals flake, developers ignore the whole suite. The gate becomes theater.
We made flakiness a first-class bug. A flaky eval gets a ticket, a deadline, and an owner. We tracked the flake rate the way we tracked the pass rate. A suite you cannot trust is worse than no suite at all.
Trustworthiness: Transparency About What This System Does and Doesn't Do
No eval system is perfect. Pretending otherwise is how trust dies. Here is the honest accounting.
What this system catches well: mechanical failures, regressions against golden data, format and schema violations, and gross behavior shifts. These are the high-confidence, high-volume cases.
What it struggles with: subtle reasoning errors, adversarial inputs that look benign, and long-horizon harms that only appear after many turns. No automated suite fully solves these.
Where humans still matter: final judgment on ambiguous outputs, edge cases near thresholds, and the ongoing curation of golden data. Automation scales the review; it does not replace it.
The failure modes we watch: judge bias, stale golden data, threshold gaming, and flaky evals masking real regressions. We monitor all four explicitly.
The audit trail: every eval run is logged. Every verdict is traceable to a case, a version, and a reviewer. If a customer reports a problem, we can reconstruct exactly what the system saw and decided.
This transparency is not a weakness. It is the point. The system earns trust by being honest about its limits.
How Many Evals Do You Actually Need?
The honest answer: fewer than you think, and more than you have.
You do not need 5,000 evals on day one. You need enough to cover your highest-risk behaviors with golden data, plus fast unit tests for mechanics, plus a live sampling loop for the unknown.
Start with the golden core. Identify the ten behaviors that would be catastrophic if they broke. Write golden cases for those. Wire them into CI. Then expand outward as you learn where the model fails.
The number matters less than the coverage. 50 well-targeted golden evals beat 5,000 shallow ones. Scale the count only after the foundation is trustworthy.
How Much Do Evals Cost?
Budget for three buckets: compute, latency, and maintenance.
Compute is the token spend per run. Latency is the developer time waiting on the suite. Maintenance is the human time keeping evals from rotting. Most teams under-budget maintenance and over-budget compute.
A useful rule of thumb: tier your evals so the cheap ones run on every PR and the expensive ones run on merge and nightly. Cache unchanged evals. Sample live traffic. That keeps the bill flat while coverage grows.
Expert Q&A
Q: How do you decide whether to use a golden dataset, a unit test, or a live eval for a given behavior? A: Match the eval type to the determinism of the output. If a behavior has a single correct answer — a JSON schema, a required field, a tool call, a fixed format — use a deterministic unit test. It is fast, cheap, and never flaky. If a behavior has a known-good answer that a human can curate once — a policy question, a compliance response, a headline — use a golden dataset case. It anchors regression testing and gives you a stable source of truth. If a behavior is open-ended and only well-defined in the wild — a summarization quality, a tone judgment, a reasoning task — use a live eval sampled from production traffic and scored by an LLM-as-judge with human spot-checks. The trap is forcing every behavior into one bucket. Most teams over-index on LLM-as-judge for things a unit test would handle deterministically, which wastes money and adds flakiness. Rule of thumb: if you can assert it, assert it.
Q: What pass-rate threshold should we set, and how do we stop teams from gaming it? A: There is no universal threshold, and a single number is the wrong unit of control. Start with a threshold that reflects your risk tolerance for the specific model and feature — often 85–95% for the golden core, lower for live evals — but treat it as a floor, not a target. The moment a threshold becomes a target, developers will tune prompts to pass it rather than improve the model. Defend against this three ways. First, use adversarial evals: cases deliberately designed to break the model, so "passing" requires genuine robustness. Second, rotate and expand the golden set so the suite cannot be memorized; a static suite gets gamed by construction. Third, watch the trend, not the single number — a threshold that stays green while the distribution of scores degrades is a false signal. And never let a threshold alone authorize a merge; pair it with a human review of near-threshold and failure cases. The number is a tripwire, not the quality system.
Q: Our LLM-as-judge keeps disagreeing with our human reviewers. How do we reconcile them? A: Disagreement is expected, and you should design for it rather than fight it. First, accept that the judge is a filter, not a verdict. Its job is to narrow the set of cases humans must look at, not to make the final call. Second, measure and track the disagreement rate explicitly — if the judge and humans disagree on more than a small fraction of cases, the judge's rubric is misaligned, and you should fix the rubric, not the model. Third, calibrate the judge against a human-labeled sample: build a small, gold-standard set of human verdicts, run the judge on it, and compute precision and recall. If the judge is systematically lenient on verbosity or harsh on a particular error class, adjust the scoring prompt or switch to a rubric-based judge. Fourth, route by confidence: anything near the threshold goes to human review automatically. Finally, re-run this calibration whenever you swap the underlying model — judge bias drifts with the model it evaluates. The goal is not perfect agreement; it is a judge that reliably flags the cases that matter.
Q: How do you keep 5,000 evals from rotting, and what does "maintenance" actually cost? A: Eval rot is the silent killer of eval systems, and it is almost always a people problem, not a code problem. Golden cases go stale when product requirements shift. Thresholds drift as the model improves or degrades. Live-eval sampling becomes unrepresentative as traffic patterns change. We treat maintenance as a first-class engineering activity with an explicit owner, not a spare-time chore. Concretely: every golden case carries a last-reviewed date and an owner, and we run a scheduled review cycle that flags cases older than a threshold. We track flake rate as a first-class metric — a flaky eval gets a ticket, a deadline, and an owner, because one flaky eval erodes trust in the whole suite. We measure coverage against the shared rubric so we know what behaviors are untested. Budget-wise, plan for maintenance to be a meaningful fraction of your QA team's time — in our experience it is often 20–30% of the pod's capacity, which is more than most teams allocate. If you under-budget maintenance, the suite decays into noise and the gate becomes theater. A maintained 500-case suite is worth more than an unmaintained 5,000-case one.
Q: We're a small team and can't build all this infrastructure. What's the minimum viable version? A: Start far smaller than you think, and resist the urge to build the full platform on day one. The minimum viable system is four things. One: a small golden dataset — even 20–50 human-curated cases covering your highest-risk behaviors, each with a known-good answer. Two: deterministic unit tests for anything mechanical — JSON validity, required fields, tool-call correctness, format. Three: a script that runs all of it and outputs a pass/fail and a list of failing cases. Four: a CI hook that runs that script on every PR and blocks the merge on failure. That is genuinely enough to start, and it is maybe a few days of work. From there, add live evals with an LLM-as-judge only once your golden core is stable and you have the capacity to spot-check judge output — adding the judge too early just adds flakiness and cost. Tier your evals so cheap ones run on every PR and expensive ones run on merge and nightly. And appoint one person to own the suite's health from the start, because rot begins the day nobody owns it. You do not need 5,000 evals to be trustworthy; you need a small, maintained, honest core that actually gates your deploys.
Q: How do we ensure our eval suite is actually independent and not just validating whatever the model team wants? A: Independence is an organizational property, not a technical one, and it has to be designed deliberately. The single highest-leverage move is the reporting line: our QA function reports to platform engineering, not to product or data science. That removes the incentive to make the eval pass to ship a feature. The second move is ownership separation: product pods own their domain evals, but the central LLM QA team owns the harness, the golden datasets, and the shared infrastructure. No single team can both define the test and be the only one who can change it. The third move is the shared rubric: a single, written definition of "acceptable output" that every eval traces back to, so quality is not whatever the feature team happens to think today. The fourth is the audit trail: every run is logged, every verdict is traceable to a case, a version, and a reviewer. If a model ships and fails, you can reconstruct exactly what the system saw and decided. Independence is what makes the gate meaningful — without it, you have a rubber stamp with a pass rate.
About the Author
The author is the QA Platform Lead on the Platform Engineering team, with 15 years in software testing and 6 years building LLM evaluation systems. All figures in this article come from internal run logs and cost reports. The views expressed are their own and based on direct operational experience.