← All AI/ML open problems

Optimization & optimizers · Learning dynamics & scaling / Open question

How Much Optimizer Memory Is Necessary?

Large models devote substantial memory to optimizer statistics. A theory should explain when those statistics can be compressed almost for free and when the lost geometry matters. Efficiency payoff: Compress optimizer state with a quantified convergence and risk cost.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

For stochastic factorization L(U,V)=12∥UV⊤−M⋆∥F2L(U,V)=\tfrac12\|UV^\top-M_\star\|_F^2 with rank-rr target, compare clipped Adafactor with matched full elementwise-second-moment updates. For each factor's positive, smoothed second-moment matrix StS_t, define F(S)=S11⊤S/(1⊤S1)\mathcal F(S)=S\mathbf1\mathbf1^\top S/(\mathbf1^\top S\mathbf1) and assume ∥St−F(St)∥F≤η∥St∥F\|S_t-\mathcal F(S_t)\|_F\leq\eta\|S_t\|_F, with bounded drift along either trajectory. Determine the optimizer-memory versus convergence frontier in r,ηr,\eta, target conditioning and drift, retaining feedback between approximate preconditioning and later gradients.

Assumptions and scope

  • Fix the precise Adafactor variant: row/column accumulators, denominator smoothing, decay schedule, update clipping, relative-step scaling and optional first momentum.
  • Use the same first-moment, step-size and clipping conventions in the full-accumulator comparator; separate a matrix of elementwise second moments from the full gradient covariance.
  • Declare an objective/noise family and a measurable factorization-error condition. Do not assume the adaptive gradient stream is independent of the approximate preconditioner.

What would settle it?

A trajectory or convergence comparison with explicit dependence on nonseparability, drift and memory, plus a lower-bound family or an adaptive-rank correction attaining the trade-off.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Adafactor's factorization and stabilization mechanisms are established. Factored optimizers such as SMMF already have regret analyses. The residual target is a sharp comparison for the specified clipped, time-varying Adafactor dynamics, not the existence of any memory-efficient convergence result. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Shazeer and Stern — Adafactor: Adaptive Learning Rates with Sublinear Memory Cost ↗related algorithm · source checked 2026-11-19

    Primary search abstract checked: row/column second-moment factorization, update clipping, increasing decay and relative-step scaling. The original memory-saving mechanism is known.

  2. Park and Lee — SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization ↗partial theory · source checked 2026-11-19

    Primary search abstract checked: factored first/second moments with a regret analysis compared to AdamNC. This is evidence that memory-efficient optimizer theory exists, not a convergence theorem for every Adafactor variant.

  3. Cohen et al. — Adaptive Gradient Methods at the Edge of Stability ↗empirical motivation · source checked 2026-11-19

    Abstract checked: empirical adaptive edge of stability; the preconditioned-Hessian top eigenvalue equilibrates near 38/η for Adam with β1=0.9, and Adam keeps entering high-curvature regions.