← All AI/ML open problems

Generative models & sampling · Learning dynamics & scaling · Transformer theory / Open question

When Does Denoising Learn a Distribution?

Denoising can generate strikingly realistic samples, yet minimizing empirical denoising loss can also lead to memorization. A learning theorem must explain the difference.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

For K-component spherical Gaussian mixtures in R^d with known common variance, component masses at least pi_min, bounded means and separation at least Delta, train a two-layer ReLU denoiser on a reused dataset using Gaussian denoising score matching. Establish a finite-sample SGD and discretized reverse-diffusion guarantee in Wasserstein distance, jointly accounting for width, optimization time, noise cutoff and sampling steps. Distinguish population recovery from reproducing the finite training sample.

Assumptions and scope

  • Fix mixture or latent structure, conditioning, minimum component mass and dimensionality. General efficient score estimation can face computational lower bounds.
  • Specify the neural architecture, denoising loss, noise schedule, optimizer, stopping rule and data reuse; do not replace the empirical objective by population access.
  • Choose a distributional metric and charge both training and sampling compute. State early-stopping or smoothing bias, especially for singular target distributions.

What would settle it?

A finite-sample, finite-compute population-generation bound with an implementable training/regularization rule for the declared multimodal family, plus a sharp failure or memorization regime.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Score-to-sampling reductions, structured mixture-learning algorithms, population transformer-denoiser convergence and reverse-Fisher gradient guarantees already exist. Empirical-score memorization has also been analyzed. The remaining target is a joint finite-data neural-training and sampling guarantee under explicit structure. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Chewi et al. — DDPM Score Matching and Distribution Learning ↗partial theory · source checked 2026-11-19

    Publisher abstract checked: reductions to parameter/density learning, minimax rates in structured classes and computational lower bounds for general families. Assuming a learned score is accurate omits a central difficulty.

  2. Chen, Kontonis and Shah — Learning General Gaussian Mixtures with Efficient Score Matching ↗partial theory · source checked 2026-11-19

    Publisher abstract checked: a score-learning sampler with d^poly(k/epsilon) complexity under bounded conditioning and parameter radius, without separation. This is not a general polynomial-in-all-parameters neural-SGD guarantee.

  3. Li, Min and Vidal — Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs ↗partial theory · source checked 2026-11-19

    Primary abstract checked: population-DDPM gradient convergence and token/iteration bounds in a multi-token mixture model. The finite-dataset and complete sampling pipeline remain separate issues.

  4. Tyurin — Global Convergence of Gradient Descent for Score Matching in Gaussian Mixtures via Reverse Fisher Divergence ↗partial theory · source checked 2026-11-19

    Primary abstract checked: global gradient-descent guarantees for a reverse-Fisher objective under specified Gaussian-mixture assumptions. It is a different objective from ordinary empirical forward denoising score matching.

  5. Baptista et al. — Memorization and Regularization in Generative Diffusion Models ↗partial theory · source checked 2026-11-19

    Primary abstract checked: exact empirical-score minimization can memorize training samples; regularization is needed, with several mechanisms investigated. Zero empirical denoising error is not a population-generation guarantee.