Why Two Identical AI Training Runs Can Still Disagree

Two training jobs start from the same checkpoint, use the same seed and follow what looks like the same recipe. Hours later, their printed losses agree to four decimal places, but the resulting checkpoints are different. That is not necessarily evidence of a bug. It is a reminder that “same run” has more conditions than a seed can capture.

Reproducibility has more than one meaning

NVIDIA’s Megatron Core discussion separates independent reproducibility from bitwise checkpoint resume. Independent runs ask whether repeating a recipe under defined conditions reproduces the result; bitwise determinism asks whether the numerical state is exactly identical, including after resuming from a checkpoint. The scope is tied to a fixed recipe, hardware, software and parallelism. The vendor’s overhead examples are configuration-specific, not universal cost estimates. (NVIDIA Megatron Core)

This distinction helps teams define what they need. A research group may care that repeated experiments support the same conclusion within expected variation. A debugging team may need an exact replay so it can compare a code change with a known baseline. A recovery system may need checkpoint resume to continue without introducing a new numerical path. These are related goals, but they require different evidence.

Question	Reproducibility concern	Useful evidence
Can a separate run repeat the experiment?	Recipe and environment consistency	Recorded versions, configuration and repeated outcomes
Can a resumed job continue identically?	Checkpoint and operation determinism	Exact comparisons after resume under the fixed setup
Did the model improve?	Quality on intended tasks	Evaluation beyond numerical equality
Did two visible losses match?	Rounded reporting may hide differences	Intermediate tensor or checkpoint fingerprints

Bitwise equality is a strong debugging property, not a proxy for model quality. Two identical runs can both be poor, and two runs that differ slightly may still support the same scientific conclusion. Teams should state which question their reproducibility claim answers.

Floating-point arithmetic remembers the order

In ordinary arithmetic, addition is associative: grouping terms differently gives the same result. Floating-point arithmetic rounds intermediate values, so changing the order can change the final value. Here is an intentionally small conceptual illustration using values with very different magnitudes:

(large_value + negative_large_value) + small_value
large_value + (negative_large_value + small_value)

Depending on the precision and rounding, the small contribution may disappear in one grouping and survive differently in another. Real training uses far more operations, and parallel systems may combine partial results in different orders. The point is not that one grouping is always wrong; it is that a mathematically equivalent calculation can produce a distinct finite-precision result.

Once a tiny difference enters an update, the next step starts from slightly different weights. Later calculations can preserve, dampen or amplify that difference. A run can therefore remain close in a broad metric while no longer being bitwise identical. The specific path depends on the model and computation; there is no universal rule that every small numerical difference will grow dramatically.

A seed is only one part of the recipe

Random seeds control selected random choices, such as initialization or data ordering, where the framework makes them deterministic. But deterministic behavior also depends on which kernels run, how operations are parallelized and how partial sums are combined. Software versions can change implementation details. Hardware may use different numerical behavior or operation paths. A resume may restore some state but omit another source of progress unless the checkpoint captures it.

That is why “same seed” is an incomplete reproducibility report. A team investigating divergence should record the exact software stack, hardware, parallelism, configuration and checkpoint. It should decide whether the supported claim is limited to one fixed configuration or expected to cover a broader set. The broader the claim, the more environments and cases require evidence.

Changing parallelism is especially relevant. Splitting a tensor operation across more devices changes how partial work is grouped and reduced. Even if every device receives the same conceptual inputs, a different topology can alter summation order. A team seeking bitwise identity should treat topology as part of the recipe, not as a transparent capacity setting.

Find the first divergence instead of comparing the final loss

Rounded losses are useful summaries for training progress, but they are lossy summaries. Two values can print identically after rounding even though their underlying numbers differ. A final loss also cannot reveal when the paths separated. To debug, compare intermediate tensors at consistent points in execution and use ordered fingerprints that preserve which tensor and operation stage each result represents.

A diagnostic sequence might compare the restored checkpoint first, then the next data batch, then selected intermediate activations and gradients, and finally updated weights. If the checkpoint already differs, the issue precedes the resumed step. If the checkpoint matches but an intermediate tensor diverges, narrow the search to the operation that produced it. A fingerprint helps identify the first mismatch efficiently; it does not explain the cause on its own.

Care is needed when collecting comparisons. Tensor ordering, device placement and checkpoint boundaries must be consistent, otherwise the diagnostic creates noise. Fingerprints should be associated with clear names and step identifiers. Teams also need to decide which tensors are informative enough to inspect without turning every training run into a storage-heavy tracing exercise.

Determinism is a debugging tool, not the finish line

For a training platform, reproducibility makes regressions easier to isolate and checkpoint recovery more trustworthy. If two runs match under a fixed setup, a developer can focus on the code change rather than unexplained numerical drift. If they diverge, ordered comparisons can identify the point where investigation should begin.

There can be a cost. Deterministic operation choices may constrain performance or available kernels in some configurations. NVIDIA presents particular overhead case studies, but those figures belong to their stated setups. A team should measure its own trade-off and decide where exact replay is essential: every development run, only release candidates, or a targeted debugging mode.

Reproducing a run and choosing a meaningful exam are complementary questions. The double-blind evaluation pilot addresses test confidentiality, while SWE-Serve’s live checks show how the verifier changes what counts as a successful software change.

The most useful engineering practice is precise language. Say whether a claim concerns repeated outcomes, exact checkpoint resume or evaluation consistency. Keep environment details with the run. When debugging, compare internal states rather than trusting rounded displays. That turns reproducibility from a vague promise into a testable property—and makes disagreement between runs a clue that can be investigated instead of a mystery.