← All AI/ML open problems

Optimization & optimizers · Learning dynamics & scaling · Transformer theory / Open question

When Does Muon Beat Adam?

Muon and Adam treat gradient geometry differently. Explaining when that difference helps could guide optimizer choice beyond trial-and-error benchmarks.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Study simultaneous training of U and V in a two-layer factorized linear predictor x mapped to UV-transpose x, with Gaussian inputs of bounded covariance condition number, a rank-r teacher and Gaussian label noise. Compare momentum Muon with a fixed finite Newton–Schulz implementation against Adam under matched samples and charged arithmetic. Determine geometry-dependent regimes of faster, equivalent or slower convergence to population excess risk epsilon, including dependence on factor rank, basis orientation and conditioning.

Assumptions and scope

  • Specify each optimizer completely, including momentum, stabilization, weight decay, learning-rate schedules and Muon's finite Newton–Schulz polynomial and iteration count.
  • Compare equal data access and explicit arithmetic costs, including orthogonalization; distinguish ideal polar updates from practical finite-precision approximations.
  • Fix a task and architecture family with meaningful rank/conditioning parameters. No optimizer is assumed uniformly superior over arbitrary reparameterizations or objectives.

What would settle it?

A proved separation or near-equivalence with explicit geometry-dependent regimes under matched resources, including a boundary example and sensitivity to finite-step orthogonalization.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Muon already has stabilized finite-Newton–Schulz last-iterate results and cycling/noise counterexamples. A November 2026 paper analyzes staged transformer next-token training and spectral margins. The target is a comparative simultaneous nonlinear learning guarantee, not first convergence of practical Muon. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Wu et al. — Theoretical Analysis of Muon on Transformers for Next-Token Prediction ↗partial theory · source checked 2026-11-19

    Primary abstract checked: staged output-value then key-query training, spectral-margin and nonasymptotic softmax-attention guarantees, with generalization to unseen contexts. Simultaneous multi-layer training is not covered by this abstract-level check.

  2. Li and Tan — Last-Iterate Stationarity of Muon with Finitely Many Newton–Schulz Iterations ↗partial theory · source checked 2026-11-19

    Primary abstract checked: stabilization-dependent deterministic convergence, stochastic convergence under schedules, cycling/noise counterexamples and limits on pointwise polynomial rates. Finite-iteration Muon convergence is already studied.

  3. Cohen et al. — Adaptive Gradient Methods at the Edge of Stability ↗empirical motivation · source checked 2026-11-19

    Abstract checked: empirical adaptive edge of stability; the preconditioned-Hessian top eigenvalue equilibrates near 38/η for Adam with β1=0.9, and Adam keeps entering high-curvature regions.