Optimization & optimizers · Learning dynamics & scaling / Open question
How Much Optimizer Memory Is Necessary?
Large models devote substantial memory to optimizer statistics. A theory should explain when those statistics can be compressed almost for free and when the lost geometry matters. Efficiency payoff: Compress optimizer state with a quantified convergence and risk cost.
Seed catalog · Showing the bundled record.
Problem statement
For stochastic factorization with rank- target, compare clipped Adafactor with matched full elementwise-second-moment updates. For each factor's positive, smoothed second-moment matrix , define and assume , with bounded drift along either trajectory. Determine the optimizer-memory versus convergence frontier in , target conditioning and drift, retaining feedback between approximate preconditioning and later gradients.
Assumptions and scope
- Fix the precise Adafactor variant: row/column accumulators, denominator smoothing, decay schedule, update clipping, relative-step scaling and optional first momentum.
- Use the same first-moment, step-size and clipping conventions in the full-accumulator comparator; separate a matrix of elementwise second moments from the full gradient covariance.
- Declare an objective/noise family and a measurable factorization-error condition. Do not assume the adaptive gradient stream is independent of the approximate preconditioner.
What would settle it?
A trajectory or convergence comparison with explicit dependence on nonseparability, drift and memory, plus a lower-bound family or an adaptive-rank correction attaining the trade-off.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Adafactor's factorization and stabilization mechanisms are established. Factored optimizers such as SMMF already have regret analyses. The residual target is a sharp comparison for the specified clipped, time-varying Adafactor dynamics, not the existence of any memory-efficient convergence result. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Shazeer and Stern — Adafactor: Adaptive Learning Rates with Sublinear Memory Cost ↗related algorithm · source checked 2026-11-19
Primary search abstract checked: row/column second-moment factorization, update clipping, increasing decay and relative-step scaling. The original memory-saving mechanism is known.
- Park and Lee — SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization ↗partial theory · source checked 2026-11-19
Primary search abstract checked: factored first/second moments with a regret analysis compared to AdamNC. This is evidence that memory-efficient optimizer theory exists, not a convergence theorem for every Adafactor variant.
- Cohen et al. — Adaptive Gradient Methods at the Edge of Stability ↗empirical motivation · source checked 2026-11-19
Abstract checked: empirical adaptive edge of stability; the preconditioned-Hessian top eigenvalue equilibrates near 38/η for Adam with β1=0.9, and Adam keeps entering high-curvature regions.