Synthetic Data and Model Collapse: Why Verification Helps but Is Not Enough

Conceptual architecture; arrows show relationships, not measured latency, energy or safety guarantees.

What model-collapse experiments show

Shumailov and colleagues studied how successive training on model-generated data can distort the learned distribution. Rare events can disappear as approximation and sampling errors accumulate. The finding concerns a training process, not a biological disease or proof that every use of synthetic text fails after five generations. ^[1]

A table assigning a fixed variance to generations one through five would need a defined experiment. There is no universal timetable from diverse language to gibberish that applies to every model and dataset.

Replacement and accumulation differ

A separate study found that retaining original real data while accumulating synthetic generations avoided collapse in its tested settings. That is important counterevidence to the claim that formal self-play is the only mathematically viable solution. It is also an experimental result with conditions, not permission to ignore data quality. ^[2]

Track source provenance, duplication, coverage and distribution shifts. A filter that rewards fluent prose may miss factual errors, while an overly narrow filter can remove useful diversity. An LLM judge is one signal whose behavior needs evaluation, not an oracle or an inevitably harmful component.

External verification improves the feedback

AlphaProof trains on formal mathematics using Lean. That provides a much stronger correctness signal for a stated formal proposition than judging whether an answer sounds convincing. DeepMind’s 2024 announcement also describes the specific role of AlphaGeometry 2 and the evaluation conditions; it is not a claim that one system solved all IMO problems autonomously. ^[3]

Formalization still matters. A proof can establish the wrong proposition if the original problem was translated incorrectly or the specification omitted a requirement.

A passing test is narrower than ground truth

For generated software, execution and tests can reject many bad candidates. They cannot prove behavior on every untested input, guarantee freedom from vulnerabilities or certify performance outside the measured environment.

Use separate held-out evaluations, track leakage and review how the tests were produced. If the same model generates a problem, its solution and a weak test, those pieces can agree while sharing an error.

A dependable synthetic-data pipeline asks what each verifier actually checks, which cases it misses and how diversity is preserved. There is no need to declare human data exhausted or promise hallucination-free self-improvement to explain why better feedback is valuable.