← All AI/ML open problems

Learning dynamics & scaling · Optimization & optimizers / Open question

Learning at the Edge of Stability

Successful training often operates beyond the small-step regime of classical descent proofs. Adaptive optimizers change that regime as they learn. Efficiency payoff: Use larger productive steps without unstable runs or unnecessarily conservative schedules.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Study minibatch AdamW on two-layer tanh networks fitting a noisy two-index Gaussian regression teacher, with bounded teacher coefficients and both network layers trained. Vary width, step size, moment-decay coefficients and weight decay near the preconditioned Hessian's stability boundary. Derive a finite-time approximation jointly tracking weights, momentum and the adaptive second moments, and a population-risk consequence that distinguishes stable oscillatory learning from divergence or noise fitting.

Assumptions and scope

  • Fix the loss, initialization, step size, both moment coefficients, denominator smoothing, bias correction and decoupled weight decay.
  • State local regularity, noise and curvature-subspace assumptions; multiple nearly equal top eigenvalues and a rotating preconditioner cannot be treated as a fixed scalar curvature.
  • Use a declared population-risk or function-space quantity, accounting for parameter rescalings. Flatness in one parameterization alone is not a generalization theorem.

What would settle it?

A stochastic trajectory-coupling or averaged-dynamics theorem with an explicit validity horizon, stability region and testable implicit-bias consequence, together with failure cases outside that region.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Full-batch self-stabilization, stochastic SGD coupling and momentum-dependent thresholds already exist. Adaptive-edge observations and Adam rod-flow models also exist. The remaining target is controlled minibatch AdamW dynamics with an evolving multidirectional preconditioner and a justified function-space consequence. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Damian, Nichani and Lee — Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability ↗partial theory · source checked 2026-11-19

    Abstract checked: full-batch gradient descent at the edge of stability tracks projected gradient descent with sharpness at most 2/η under stated conditions.

  2. Liao, Kolomvaki and Kyrillidis — SGD at the Edge of Stability: The Stochastic Sharpness Gap ↗partial theory · source checked 2026-11-19

    Abstract checked: stochastic self-stabilization and a coupling theorem giving a closed-form sharpness gap below 2/η for SGD.

  3. Andreyev et al. — Momentum Further Constrains Sharpness at the Edge of Stochastic Stability ↗empirical motivation · source checked 2026-11-19

    Abstract checked: batch-size-dependent sharpness plateaus for SGD with momentum, mostly empirical, aligned with linear-stability thresholds.

  4. Cohen et al. — Adaptive Gradient Methods at the Edge of Stability ↗empirical motivation · source checked 2026-11-19

    Abstract checked: empirical adaptive edge of stability; the preconditioned-Hessian top eigenvalue equilibrates near 38/η for Adam with β1=0.9, and Adam keeps entering high-curvature regions.

  5. Regis and Chewi — A Rod Flow Model for Adam at the Edge of Stability ↗related result · source checked 2026-11-19

    Abstract checked: continuous-time rod-flow models for Adam and other momentum/adaptive optimizers, evaluated empirically rather than proved to track the iterates.

  6. Damian, Ma and Lee — Label Noise SGD Provably Prefers Flat Global Minimizers ↗partial theory · source checked 2026-11-19

    Abstract checked: label-noise SGD tracks an explicitly regularized objective, with extensions to momentum and general noise covariance; this is existing implicit-bias theory.