Optimization & optimizers · Learning dynamics & scaling · Transformer theory / Open question
When Does Muon Beat Adam?
Muon and Adam treat gradient geometry differently. Explaining when that difference helps could guide optimizer choice beyond trial-and-error benchmarks.
Seed catalog · Showing the bundled record.
Problem statement
Study simultaneous training of U and V in a two-layer factorized linear predictor x mapped to UV-transpose x, with Gaussian inputs of bounded covariance condition number, a rank-r teacher and Gaussian label noise. Compare momentum Muon with a fixed finite Newton–Schulz implementation against Adam under matched samples and charged arithmetic. Determine geometry-dependent regimes of faster, equivalent or slower convergence to population excess risk epsilon, including dependence on factor rank, basis orientation and conditioning.
Assumptions and scope
- Specify each optimizer completely, including momentum, stabilization, weight decay, learning-rate schedules and Muon's finite Newton–Schulz polynomial and iteration count.
- Compare equal data access and explicit arithmetic costs, including orthogonalization; distinguish ideal polar updates from practical finite-precision approximations.
- Fix a task and architecture family with meaningful rank/conditioning parameters. No optimizer is assumed uniformly superior over arbitrary reparameterizations or objectives.
What would settle it?
A proved separation or near-equivalence with explicit geometry-dependent regimes under matched resources, including a boundary example and sensitivity to finite-step orthogonalization.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Muon already has stabilized finite-Newton–Schulz last-iterate results and cycling/noise counterexamples. A November 2026 paper analyzes staged transformer next-token training and spectral margins. The target is a comparative simultaneous nonlinear learning guarantee, not first convergence of practical Muon. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Wu et al. — Theoretical Analysis of Muon on Transformers for Next-Token Prediction ↗partial theory · source checked 2026-11-19
Primary abstract checked: staged output-value then key-query training, spectral-margin and nonasymptotic softmax-attention guarantees, with generalization to unseen contexts. Simultaneous multi-layer training is not covered by this abstract-level check.
- Li and Tan — Last-Iterate Stationarity of Muon with Finitely Many Newton–Schulz Iterations ↗partial theory · source checked 2026-11-19
Primary abstract checked: stabilization-dependent deterministic convergence, stochastic convergence under schedules, cycling/noise counterexamples and limits on pointwise polynomial rates. Finite-iteration Muon convergence is already studied.
- Cohen et al. — Adaptive Gradient Methods at the Edge of Stability ↗empirical motivation · source checked 2026-11-19
Abstract checked: empirical adaptive edge of stability; the preconditioned-Hessian top eigenvalue equilibrates near 38/η for Adam with β1=0.9, and Adam keeps entering high-curvature regions.