Transfer & distribution shift · Statistical learning theory / Open question
Self-Training Without Collapse
Training data increasingly contain model-generated content. Retaining average quality may still conceal the gradual loss of rare knowledge.
Seed catalog · Showing the bundled record.
Problem statement
Start from K spherical Gaussian components in d dimensions with common known variance, separation at least Delta and masses at least pi_min. At each generation fit a K-component mixture to at most M retained real/synthetic samples; an adaptive filter has per-component acceptance probability at least q_min. Determine the fresh-real-data and replay budgets needed to keep every component's recovered mass and mean within epsilon over T generations, including unknown component assignments, filtering bias and finite optimization error.
Assumptions and scope
- Fix mixture separation, minimum mode mass and the retained-data budget; define collapse through a stated distributional or rare-mode error threshold.
- Specify replacement, accumulation or compression exactly, and charge fresh real samples. Unlimited accumulation is a different resource model.
- Declare the filter's observables and accepted-sample probabilities. Treat arbitrary unknown filtering bias as a potential identifiability obstruction.
What would settle it?
Matching long-horizon real-sample and memory requirements, with a retention/filtering procedure attaining them or a lower bound proving unavoidable loss in an identified regime.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Accumulation can avoid collapse in linear and iterative-MLE models, but counterexamples also exist. Model-agnostic contaminated-training bounds and a sharp uncurated Gaussian criterion have been proposed. The target adds bounded memory, adaptive selection and rare-mode preservation; it does not claim collapse is inevitable. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Gerstgrasser et al. — Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data ↗partial theory · source checked 2026-11-19
Abstract checked: accumulating data avoids collapse empirically and in a linear-regression framework with a finite error bound.
- Barzilai and Shamir — When Models Don't Collapse: On the Consistency of Iterative MLE ↗partial theory · source checked 2026-11-19
Abstract checked: non-asymptotic no-collapse bounds for iterative MLE with accumulating data, and examples where collapse is arbitrarily fast without extra assumptions.
- Turgeman and Tirer — Model Collapse Is Avoidable Even as Real Data Vanishes ↗partial theory · source checked 2026-11-19
Abstract checked: a sharp consistency criterion for iterative Gaussian estimation with uncurated real/synthetic mixtures, including vanishing real fractions.
- Wang, Niu and Li — Can Generative AI Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training ↗partial theory · source checked 2026-11-19
Abstract checked: model-agnostic convergence with rate given by the minimum of intrinsic rate and real-data fraction; bias correction.