← All AI/ML open problems

Learning dynamics & scaling · Transformer theory / Open question

Why Neural Networks Grok

A network can memorize quickly yet discover a general rule much later. Predicting that delay would explain a striking disconnect between training success and real learning.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Train a finite-width two-block causal softmax transformer on a uniformly sampled subset of pairs (a,b)∈Zp2(a,b)\in\mathbb Z_p^2, labelled by a+b mod pa+b\bmod p, for prime pp, using cross-entropy and minibatch AdamW. Define fitting and generalization times by fixed accuracy thresholds on the training subset and the uniform population. Prove or refute a regime Ttrain≪TgenT_{\mathrm{train}}\ll T_{\mathrm{gen}}, with explicit dependence on pp, sample fraction, width, initialization, learning rate and weight decay.

Assumptions and scope

  • Fix the modulus, token encoding, architecture, training-set distribution and full-domain test risk; define both times by explicit thresholds and persistence conditions.
  • Specify the optimizer, batch reuse and whether regularization is coupled or decoupled. The model must retain nonlinear attention and feature evolution.
  • Control randomness and finite-size approximation errors over the full delayed interval; a fitted timescale or an assumed final generalizing solution is insufficient.

What would settle it?

An end-to-end finite-sample delay theorem and a parameter-dependent phase boundary for the declared transformer, with both early poor generalization and eventual improvement proved.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Ridge-regression grokking is rigorous, but nonlinear results also already exist: Lyu et al. analyze homogeneous networks, Mohamadi et al. analyze quadratic modular-addition models, and Pracher et al. derive spectral delay mechanisms. The extension here is quantitative minibatch softmax-transformer dynamics, not the first nonlinear example. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Xu, Vardi and Safran — To Grok Grokking: Provable Grokking in Ridge Regression ↗partial theory · source checked 2026-11-19

    Abstract checked: end-to-end linear grokking and quantitative hyperparameter-dependent delay bounds; nonlinear extensions in this paper are empirical.

  2. Lyu et al. — Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking ↗partial theory · source checked 2026-11-19

    Abstract checked: rigorous grokking for homogeneous neural networks with large initialization and small weight decay. Nonlinear grokking is therefore not an open existence question.

  3. Mohamadi et al. — Why Do You Grok? A Theoretical Analysis of Grokking Modular Addition ↗partial theory · source checked 2026-11-19

    Abstract checked: kernel-regime lower bounds and generalization of bounded-norm two-layer quadratic networks; gradient descent with a specified infinity-norm regularizer finds suitable solutions.

  4. Pracher et al. — A Spectral Theory of Grokking: Weight Decay Induces Feature Learning ↗partial theory · source checked 2026-11-19

    Abstract checked: homogeneous squared-loss networks, a reduced spectral dynamical system and empirical transformer phase diagrams. Quantitative delay mechanisms already exist in restricted settings.