The Reproducibility Gap: Why Your ML Pipeline Won't Run Twice

The Retrain That Never Lands Where It Started

A fraud model posts strong precision in Q2. Six months later, the team tries to retrain it for a new market and cannot get within three points of the original metrics. The code is intact. The notebooks are intact. Nobody can explain the gap.

This is the reproducibility gap, and it is not an edge case. It is the default outcome for ML systems built without deliberate controls. The model was never a single artifact; it was a snapshot of data, dependencies, hardware, evaluation choices, and human decisions at one moment in time. Change any of them and the snapshot dissolves.

The instinct is to treat this as a discipline problem — hire more careful engineers, write stricter process docs. That framing fails. The reproducibility gap is a systems problem, and it widens predictably as teams scale, hand off models, and rotate personnel. The fix is equally systemic: versioning, pinning, tracking, and governance treated as one connected practice rather than four separate chores.

This article breaks down what actually breaks, what it costs, and how to build a machine learning pipeline that runs twice. (See also: ML pipeline versioning and governance best practices)

What the Reproducibility Gap Actually Is

Two terms get conflated constantly, and the distinction matters for how you design controls. The ACM's artifact-review taxonomy draws the line precisely:

Reproducibility means the same team, given the same inputs, code, and environment, gets the same result.
Replicability means a different team, given new data and an independent implementation, reaches the same conclusion.

Most organizations claim the second and cannot deliver the first. The gap is the distance between what your pipeline claims to do and what it can actually demonstrate on demand. It rarely announces itself. It surfaces during an audit, a model refresh, or a handoff — always at the worst time.

Five Pillars Every ML Result Depends On

Every ML result rests on five pillars, and each fails independently:

Data — the exact training and evaluation datasets, including every transformation applied between raw source and model input.
Code — the training script, preprocessing logic, feature definitions, and the specific commit that produced the run.
Environment — the operating system, Python version, library versions, CUDA and driver versions, and hardware configuration.
Randomness — every seed, sampling decision, and source of nondeterminism, controlled or documented.
Evaluation protocol — the train/test split, leakage controls, metric definitions, and thresholds used to judge the result.

Miss any one and you have a result you cannot defend. Most teams control one or two. The evaluation pillar is the one teams skip most often, and it is the most common cause of the "can't get within three points" scenario in the opening: the model reproduced fine, but the split or the metric definition quietly changed underneath it. (See also: Data versioning for reproducible ML)

Why "It Worked on My Machine" Persists in ML

Traditional software CI rests on a clean assumption: code plus a fixed environment produces deterministic output. Tests either pass or fail, and the build is reproducible by construction.

ML breaks that assumption at the foundation. Data is an input, and data changes without anyone touching the repository. Environments drift through transitive dependencies. Results are stochastic by design. A passing test proves almost nothing about whether a model will reproduce, because the model's behavior depends on artifacts that live outside version control. The CI mindset was never built for this, which is why teams carry the habit without carrying the protection.

The Five Failure Points That Break ML Pipelines

In practice, reproducibility failures cluster into five recurring patterns. Diagnose which ones you have before buying tooling.

Floating Dependencies and Silent Upgrades

A requirements.txt that says torch>=2.0 is not a specification; it is a wish. Unpinned libraries drift, transitive dependencies shift underneath you, and a routine pip install can change numerical behavior without a single line of your code changing. CUDA and driver mismatches add a second layer: the same container can produce different results on different GPU hosts.

The concrete fix is two-part. First, commit a lockfile that pins the full transitive dependency graph — pip-compile/uv/poetry.lock or conda-lock — not just your top-level requirements. Second, pin the base image by digest (image@sha256:...), because a :latest tag is itself a floating dependency that can change between builds. Pin at the artifact level, not the top level, or accept that your environment is a moving target. (See also: Dependency pinning and environment reproducibility)

Untracked Data Changes

Source tables get refreshed. A schema shifts a column type. An upstream team quietly changes a join key. Preprocessing logic lives in a notebook cell that someone edited and never committed. None of this appears in your git history, and all of it changes your model. Data is the largest and least-controlled input in most ML systems, and it is usually the first place to look when results will not reproduce.

Hidden Nondeterminism (Seeds, Hardware, Parallelism)

Unseeded random number generators are the obvious culprit, but they are rarely the whole story. The real mechanisms are specific:

Atomic-operation reductions. Kernels that use atomicAdd for gradient or reduction accumulation produce non-deterministic results because floating-point addition is not associative and thread completion order varies.
cuDNN/cuBLAS autotuning. Algorithm selection can differ across runs or hosts, changing accumulation order.
Non-deterministic backward kernels. Several common ops have deterministic forward passes but non-deterministic gradients.
Parallelism and mixed precision. Data-loader worker shuffling, thread scheduling, and mixed-precision accumulation alter results across runs.

PyTorch exposes the controls directly: torch.use_deterministic_algorithms(True) and the CUBLAS_WORKSPACE_CONFIG environment variable force deterministic kernels where they exist. Two runs with identical code and data can diverge unless determinism is explicitly enforced — and enforcement is not free. Deterministic kernels can be materially slower, and full bitwise determinism is often not achievable across different hardware. The honest goal is same-environment reproducibility, not identical numbers on every GPU in the fleet. (See also: Controlling nondeterminism in deep learning training)

Experiment Sprawl Without Tracking

Results live in notebooks, spreadsheets, Slack threads, and one engineer's memory. There is no lineage from a reported metric back to the config, commit, and dataset that produced it. When someone asks "which run gave us 0.91 AUC," the answer is a search party. Without experiment tracking, you cannot reproduce a good result even when you want to — and you cannot learn from a bad one.

The Human Handoff Gap

The original author leaves. The runbook was never written. The reason a feature was dropped, the workaround for a flaky data source, the one seed that made training stable — all tacit knowledge, all gone. Handoffs are where reproducibility fails silently, because the pipeline keeps running on momentum until the next refresh, and by then nobody can reconstruct the decisions that made it work.

What the Gap Costs

Reproducibility debt is easy to ignore because it does not show up as a line item until it does. In practice it bills you in three currencies: rework (engineers reverse-engineering their own pipeline to explain a metric), audit and regulatory exposure (a model you cannot defend is a model you cannot ship into a regulated workflow), and delayed refreshes (every retrain becomes an investigation, so models age in place while the business waits). The fraud-model team in the opening is paying all three at once.

The "Does It Run Twice?" Test

Before buying tooling, run this checklist against one production pipeline. A pipeline that passes all six is reproducible in practice; most fail three or more.

Can you name the exact commit, lockfile, and container digest that produced your current production model?
Can you reconstruct the exact training dataset — including every transformation — as of that run?
Is the evaluation protocol versioned — split logic, leakage controls, metric definitions, and thresholds?
Are seeds and determinism flags set and recorded, with the known performance cost documented?
Does every reported metric link back to a tracked run with config, data version, and code commit?
Could a new engineer retrain the model to within a stated tolerance using only committed artifacts and the runbook?

Building a Pipeline That Runs Twice

The fix is not a tool purchase; it is one connected practice. Version data and evaluation protocol alongside code. Pin the full dependency graph and the base image by digest. Set and record determinism flags, and document the throughput cost you accepted. Track every run with lineage back to config, commit, and dataset. Write the runbook as if the author is already gone — because eventually they are.

Do those five things and the reproducibility gap closes. Skip any one and you are back to hoping the snapshot holds.

Terminology note: reproducibility and replicability definitions follow the ACM artifact-review taxonomy.