Statistical learning theory · Learning dynamics & scaling · Embeddings & representations / Open question
Why Overparameterized Networks Generalize
Modern networks can fit more than their data should apparently allow. The central challenge is to predict when their learned structure makes that flexibility helpful.
Seed catalog · Showing the bundled record.
Problem statement
Let and , with unit vectors satisfying and independent Gaussian noise. Train both layers of an overparameterized two-layer ReLU network by minibatch squared-loss SGD from Gaussian initialization. Bound population risk after fitting the noisy sample, as a function of width, samples, and noise; identify a strict advantage over training only its initialized output layer or an obstruction to that advantage.
Assumptions and scope
- Fix a structured input distribution, teacher class, label-noise model and signal/nuisance geometry; arbitrary data distributions cannot share one positive generalization guarantee.
- Specify trainable layers, initialization, width, optimization time and regularization. Show that the analyzed trajectory actually learns features rather than assuming the desired representation.
- Use a matched frozen-feature comparator and separate optimization error, representation error and noise fitting; interpolation alone is not evidence of good population risk.
What would settle it?
A finite-sample training-and-risk theorem for dependent nonlinear features with a strict matched-baseline advantage, and a quantitative boundary between beneficial and harmful overfitting.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Width-independent tangent-feature bounds, arbitrary-width noisy-label SGD guarantees, label-noise implicit regularization and structured nonlinear SGD feature-learning results already exist. The scoped target couples dependent features, finite-data interpolation and noise; neither generalization nor a neural-versus-kernel separation is wholly unexplained. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Cao and Gu — Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks ↗partial theory · source checked 2026-11-19
Abstract checked: width-independent risk bounds through the neural tangent random-feature model at initialization. Generalization bounds for overparameterized networks already exist.
- Frei, Cao and Gu — Provable Generalization of SGD-Trained Neural Networks of Any Width in the Presence of Adversarial Label Noise ↗partial theory · source checked 2026-11-19
Abstract checked: one-hidden-layer leaky-ReLU SGD competes with the best halfspace for a broad distribution class. The comparator and distributional assumptions matter.
- Damian, Ma and Lee — Label Noise SGD Provably Prefers Flat Global Minimizers ↗partial theory · source checked 2026-11-19
Abstract checked: label-noise SGD tracks an explicitly regularized objective, with extensions to momentum and general noise covariance; this is existing implicit-bias theory.
- Ren et al. — Emergence and Scaling Laws in SGD Learning of Shallow Neural Networks ↗partial theory · source checked 2026-11-19
Abstract checked: rigorous online-SGD analysis of extensive-width additive single-index targets; scaling exponents in samples, steps and parameters from many emergent transitions.
- Malach et al. — Quantifying the Benefit of Using Differentiable Learning over Tangent Kernels ↗partial theory · source checked 2026-11-19
Identified as a distribution-dependent separation baseline in Feldman et al. Sections 1–2; its proof was not independently audited.