Learning dynamics & scaling · Optimization & optimizers / Open question
Learning at the Edge of Stability
Successful training often operates beyond the small-step regime of classical descent proofs. Adaptive optimizers change that regime as they learn. Efficiency payoff: Use larger productive steps without unstable runs or unnecessarily conservative schedules.
Seed catalog · Showing the bundled record.
Problem statement
Study minibatch AdamW on two-layer tanh networks fitting a noisy two-index Gaussian regression teacher, with bounded teacher coefficients and both network layers trained. Vary width, step size, moment-decay coefficients and weight decay near the preconditioned Hessian's stability boundary. Derive a finite-time approximation jointly tracking weights, momentum and the adaptive second moments, and a population-risk consequence that distinguishes stable oscillatory learning from divergence or noise fitting.
Assumptions and scope
- Fix the loss, initialization, step size, both moment coefficients, denominator smoothing, bias correction and decoupled weight decay.
- State local regularity, noise and curvature-subspace assumptions; multiple nearly equal top eigenvalues and a rotating preconditioner cannot be treated as a fixed scalar curvature.
- Use a declared population-risk or function-space quantity, accounting for parameter rescalings. Flatness in one parameterization alone is not a generalization theorem.
What would settle it?
A stochastic trajectory-coupling or averaged-dynamics theorem with an explicit validity horizon, stability region and testable implicit-bias consequence, together with failure cases outside that region.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Full-batch self-stabilization, stochastic SGD coupling and momentum-dependent thresholds already exist. Adaptive-edge observations and Adam rod-flow models also exist. The remaining target is controlled minibatch AdamW dynamics with an evolving multidirectional preconditioner and a justified function-space consequence. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Damian, Nichani and Lee — Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability ↗partial theory · source checked 2026-11-19
Abstract checked: full-batch gradient descent at the edge of stability tracks projected gradient descent with sharpness at most 2/η under stated conditions.
- Liao, Kolomvaki and Kyrillidis — SGD at the Edge of Stability: The Stochastic Sharpness Gap ↗partial theory · source checked 2026-11-19
Abstract checked: stochastic self-stabilization and a coupling theorem giving a closed-form sharpness gap below 2/η for SGD.
- Andreyev et al. — Momentum Further Constrains Sharpness at the Edge of Stochastic Stability ↗empirical motivation · source checked 2026-11-19
Abstract checked: batch-size-dependent sharpness plateaus for SGD with momentum, mostly empirical, aligned with linear-stability thresholds.
- Cohen et al. — Adaptive Gradient Methods at the Edge of Stability ↗empirical motivation · source checked 2026-11-19
Abstract checked: empirical adaptive edge of stability; the preconditioned-Hessian top eigenvalue equilibrates near 38/η for Adam with β1=0.9, and Adam keeps entering high-curvature regions.
- Regis and Chewi — A Rod Flow Model for Adam at the Edge of Stability ↗related result · source checked 2026-11-19
Abstract checked: continuous-time rod-flow models for Adam and other momentum/adaptive optimizers, evaluated empirically rather than proved to track the iterates.
- Damian, Ma and Lee — Label Noise SGD Provably Prefers Flat Global Minimizers ↗partial theory · source checked 2026-11-19
Abstract checked: label-noise SGD tracks an explicitly regularized objective, with extensions to momentum and general noise covariance; this is existing implicit-bias theory.