← All AI/ML open problems

Transfer & distribution shift · Statistical learning theory / Open question

Self-Training Without Collapse

Training data increasingly contain model-generated content. Retaining average quality may still conceal the gradual loss of rare knowledge.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Start from K spherical Gaussian components in d dimensions with common known variance, separation at least Delta and masses at least pi_min. At each generation fit a K-component mixture to at most M retained real/synthetic samples; an adaptive filter has per-component acceptance probability at least q_min. Determine the fresh-real-data and replay budgets needed to keep every component's recovered mass and mean within epsilon over T generations, including unknown component assignments, filtering bias and finite optimization error.

Assumptions and scope

  • Fix mixture separation, minimum mode mass and the retained-data budget; define collapse through a stated distributional or rare-mode error threshold.
  • Specify replacement, accumulation or compression exactly, and charge fresh real samples. Unlimited accumulation is a different resource model.
  • Declare the filter's observables and accepted-sample probabilities. Treat arbitrary unknown filtering bias as a potential identifiability obstruction.

What would settle it?

Matching long-horizon real-sample and memory requirements, with a retention/filtering procedure attaining them or a lower bound proving unavoidable loss in an identified regime.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Accumulation can avoid collapse in linear and iterative-MLE models, but counterexamples also exist. Model-agnostic contaminated-training bounds and a sharp uncurated Gaussian criterion have been proposed. The target adds bounded memory, adaptive selection and rare-mode preservation; it does not claim collapse is inevitable. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Gerstgrasser et al. — Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data ↗partial theory · source checked 2026-11-19

    Abstract checked: accumulating data avoids collapse empirically and in a linear-regression framework with a finite error bound.

  2. Barzilai and Shamir — When Models Don't Collapse: On the Consistency of Iterative MLE ↗partial theory · source checked 2026-11-19

    Abstract checked: non-asymptotic no-collapse bounds for iterative MLE with accumulating data, and examples where collapse is arbitrarily fast without extra assumptions.

  3. Turgeman and Tirer — Model Collapse Is Avoidable Even as Real Data Vanishes ↗partial theory · source checked 2026-11-19

    Abstract checked: a sharp consistency criterion for iterative Gaussian estimation with uncurated real/synthetic mixtures, including vanishing real fractions.

  4. Wang, Niu and Li — Can Generative AI Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training ↗partial theory · source checked 2026-11-19

    Abstract checked: model-agnostic convergence with rate given by the minimum of intrinsic rate and real-data fraction; bias correction.