Synthetic Data Is Eating Your Eval Set: Building Test Suites That Still Find Failures

The Eval Score Paradox

Your dashboard says 96% pass rate. Your incident channel says otherwise. Many teams now report the same uncomfortable pattern: evaluation scores climbing while production failures hold steady or rise. Those two numbers are not unrelated — they are symptoms of one problem.

Synthetic data is the culprit hiding in plain sight. LLM-generated test cases are cheap, fast, and endlessly scalable, which is exactly why they have quietly become the backbone of most evaluation suites. That is fine for coverage and training augmentation. It becomes a serious liability when synthetic cases are the primary source of your test sets, because at that point you are no longer measuring your product. You are measuring your generator.

This article offers a diagnostic framework for spotting contaminated evals and a build guide for LLM test suites that keep finding real failures. (See also: How to Build an LLM Evaluation Framework That Catches Real Failures)

Why Synthetic Data Quietly Colonized Your Eval Set

The economics were irresistible. A human-labeled edge case takes an hour of expert time and costs real money. An LLM-generated one takes seconds and costs a fraction of a cent. When you can generate thousands of test cases before lunch, hand-curating two hundred starts to look like malpractice.

The adoption curve reflects this. Synthetic generation began in training-data augmentation, where it was well understood and reasonably governed. From there it crept into prompt-engineering iteration loops — engineers generating example inputs to tune prompts — and finally into formal evaluation suites, often without a deliberate architectural decision. Most teams never voted to go synthetic-first in evaluation. It accreted, one convenient script at a time, until the synthetic share of the eval set quietly crossed the majority.

The first diagnostic step is simply to measure it. Tag every test case with its provenance — human, synthetic, trace-derived, public-benchmark — and compute the share per tier. Most teams discover the synthetic share is far higher than anyone assumed, and that it is concentrated in exactly the tiers where it does the most damage.

The feedback loop: models generating tests for models

The structural problem is sampling bias, not mysticism. A generator model produces test cases by sampling from the high-probability region of its own distribution. The system under test was trained to do well on inputs from a similar region. The result is that the generator under-samples precisely the low-probability, high-failure inputs — malformed structures, adversarial phrasing, ambiguous intent — that break the system in production. It generates tests it already knows how to pass, and the shared blind spots go untested.

You can detect this empirically rather than guessing. Run two checks:

Cross-family generation. Have a different model family generate the same test category. If the pass rate diverges sharply between same-family and cross-family generators, you have self-contamination.
Disagreement rate. Measure how often the generator and the system under test produce different outputs on the same input. A near-zero disagreement rate on a "diverse" suite is a red flag, not a green one.

The saturation problem: scores up, signal down

When your scenario and adversarial tiers plateau above ~95%, investigate. Saturation there is not a success signal — it is a test-set failure. An eval that everything passes has stopped discriminating between good and bad systems. The usual cause is that the test distribution has been optimized, implicitly or explicitly, until it matches what the system already does well.

Note the scope: a well-built unit tier should pass near 100% — those behaviors are mechanical and enumerated. Saturation is only a problem where the suite is supposed to be hard. The operational test is discrimination, not pass rate: take your current model and a deliberately degraded variant (drop a tool schema, truncate context, weaken the system prompt). A healthy eval set separates them by a wide margin. An eval set that scores both at 96% is measuring nothing.

The Three Failure Modes of Synthetic Test Sets

Not all synthetic data is bad. The failures cluster into three recognizable modes, and knowing which one you have determines the fix.

Self-contamination: when the generator and the system under test share blind spots

Self-contamination is grading your own homework. When the same model family writes the questions and answers them, their shared sampling biases mean shared failures go undetected. The eval passes not because the system is correct but because the generator cannot imagine the input that would break it. This is the most insidious failure mode because the scores look excellent and the test set looks diverse — thousands of cases, many phrasings, clean formatting. Use the cross-family and disagreement checks above to catch it.

Distribution collapse: synthetic suites are too polite

Synthetic test cases skew heavily toward clean, well-formed, solvable inputs. They have proper grammar, complete context, and clear intent. Real production traffic does not look like that. Real users are messy, adversarial, context-poor, and frequently wrong about their own problem. They paste half a spreadsheet, switch languages mid-sentence, and reference things that do not exist. A synthetic suite that never sees a malformed input will never catch a malformed-input failure. The fix is trace-derived cases (below), not better prompting of the generator — you cannot prompt your way to inputs your generator's distribution doesn't contain.

(See: Mining Production Traces for Realistic LLM Test Cases)

Benchmark leakage: public evals in the training pipeline

Public benchmarks — MMLU, GSM8K, HumanEval, SWE-bench — increasingly appear in pretraining corpora, whether scraped from GitHub, forums, or dataset mirrors. SWE-bench in particular has documented contamination concerns given its public issue/PR provenance. Decontamination efforts help at the margins but are fundamentally incomplete: you cannot reliably audit every corpus a vendor used, and a test item's influence is baked into the weights after training.

The actionable mitigations are structural, not forensic:

Treat every public benchmark score as an upper bound, not ground truth.
Prefer rolling, held-out benchmarks (dynamically refreshed sets that postdate model training cutoffs) over static leaderboards.
Embed canary strings in your private eval sets and check whether the model reproduces them — a cheap contamination tripwire for your own data.

What a Healthy Eval Set Actually Looks Like

A healthy eval set is not a single artifact. It is a layered system with explicit provenance and a deliberate, justified mix of human and synthetic origin.

The layered test suite: unit, scenario, adversarial

Structure your suite in three tiers, each with a different source and purpose:

Unit evals for deterministic behaviors: schema conformance, tool-call formatting, refusal policies, output parsing. These can be largely synthetic and auto-generated — the behaviors are mechanical and the failure modes are well enumerated. High pass rates here are expected and fine.
Scenario evals drawn from real production traces: actual user requests, redacted and curated, covering the long tail of how your product is genuinely used. These should be predominantly human-origin (curated from traces, not generated).
Adversarial evals built by a red team: deliberately hostile, out-of-distribution, and edge-case inputs designed to break the system. Never auto-generated from the system under test.

Setting the human/synthetic ratio

Resist the urge to pick a round percentage off a blog post. Set the ratio by coverage of known blind spots:

Identify the failure modes your generator demonstrably cannot produce (the cross-family disagreement check reveals these).
Require human-origin or trace-derived cases that cover each one.
Hold the line: the human-origin share of the scenario and adversarial tiers should be high enough that removing the synthetic cases would not change your pass/fail verdict.

A useful rule of thumb for teams starting out: if your scenario and adversarial tiers are majority-synthetic, you are almost certainly under-testing. Treat synthetic cases in those tiers as supplements to human-origin coverage, never as the floor.

Trace mining: the highest-ROI source you already have

Your production logs are the only source of genuinely out-of-distribution, real-world inputs. Build a lightweight pipeline:

Sample production requests, weighted toward low-confidence and error-adjacent turns.
Redact PII and secrets before anything leaves the boundary.
Cluster by intent and failure signature to avoid over-representing common paths.
Promote clusters into curated scenario cases, tagged trace-derived.

This is the single highest-leverage investment for keeping an eval set honest. It costs engineering time, not model spend, and it produces exactly the messy inputs synthetic generation cannot.

Keeping the Suite Honest Over Time

An eval set decays. Models update, distributions drift, and yesterday's hard cases become today's easy ones. Institutionalize the maintenance:

Track synthetic share per tier as a first-class metric, reviewed alongside pass rate.
Run the discrimination check (current vs. degraded model) on every eval-set change.
Rotate adversarial cases so the red team's work doesn't get memorized into the next model's training data.
Retire saturated cases from scenario/adversarial tiers and replace them with fresh trace-derived and red-team cases.

The goal is not a high score. It is a suite that still fails when your product does. If your evaluation can no longer find a failure, the most likely explanation is that it has stopped looking.