← All AI/ML open problems

Transformer theory · Embeddings & representations · Transfer & distribution shift / Open question

How Transformers Learn to Learn

A transformer being able to represent a learning algorithm does not ensure that training discovers a stable algorithm that transfers.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Let fa(x)=a1ReLU⁡(w1⊤x)+a2ReLU⁡(w2⊤x)f_a(x)=a_1\operatorname{ReLU}(w_1^\top x)+a_2\operatorname{ReLU}(w_2^\top x) with unknown shared orthonormal w1,w2w_1,w_2, x∼N(0,Id)x\sim N(0,I_d) and labels y=fa(x)+ξy=f_a(x)+\xi, ξ∼N(0,σ2)\xi\sim N(0,\sigma^2). Pretraining draws a∼P=Unif⁡([−1,1]2)a\sim P=\operatorname{Unif}([-1,1]^2). For a finite-width causal softmax transformer FθF_\theta trained by minibatch squared-loss SGD on length-mm example prompts DmD_m, bound fresh-task risk RQ(θ)=Ea∼Q,Dm,x(Fθ(Dm,x)−fa(x))2R_Q(\theta)=\mathbb E_{a\sim Q,D_m,x}(F_\theta(D_m,x)-f_a(x))^2 whenever dQ/dP≤κdQ/dP\leq\kappa, with explicit task/sample/compute dependence.

Assumptions and scope

  • The Gaussian two-feature task family and density-ratio shift are fixed in the statement. Remaining benchmark choices are token encoding, layer/head counts, normalization, initialization, learning-rate schedule and data reuse; state them before claiming a rate.
  • Distinguish existence of implementing weights, properties of population minimizers and convergence of a finite-data training algorithm. An exact query-conditioned gradient-step representation of attention alone does not establish task learning.
  • Measure expected squared error on a fresh query and a fresh test task. Report dependence on d, prompt length, training-task count, noise, width and kappa; compare to the same-observation oracle rather than assuming arbitrary distribution-shift generalization.

What would settle it?

A finite-sample training-and-transfer theorem for the declared nonlinear family, with explicit complexity/rate dependence or a lower bound separating expressivity from trainability.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Constructive approximation, linear looped-transformer convergence, stability and linear-attention subspace transfer already have theory. Huang et al. exactly represent RoPE-softmax as a query-conditioned quadratic gradient step for arbitrary weights, without proving learned task performance or transfer. Their no-go result excludes only a globally shared affine fixed-bank readout. The finite-data nonlinear training-and-transfer target remains beyond those results. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Li et al. — Transformers Meet In-Context Learning: A Universal Approximation Theory ↗partial theory · source checked 2026-11-19

    Abstract checked: constructive approximation guarantees for ICL already go beyond linear regression; construction alone is not an SGD training guarantee.

  2. Gatmiry et al. — Can Looped Transformers Learn Multi-step Gradient Descent? ↗partial theory · source checked 2026-11-19

    Abstract checked: population-loss and gradient-flow results for linear looped transformers already establish a multi-layer learnability case.

  3. Li et al. — Transformers as Algorithms: Generalization and Stability in ICL ↗partial theory · source checked 2026-11-19

    Abstract checked: stability-based in-context generalization, including dynamical data and task transfer, is established in specified regimes.

  4. Kwon et al. — Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspective ↗partial theory · source checked 2026-11-19

    Abstract checked: linear-attention conditions for out-of-distribution ICL under subspace shifts.

  5. Huang, Chlon and Chlon — RoPE Attention Is an Exact Forward-Pass Gradient Step with Softmax Intact ↗partial theory · source checked 2026-11-19

    Full HTML checked: Theorem 1/Corollary 2 give a query-conditioned effective matrix and quadratic gradient step for arbitrary weights; Proposition 3 excludes a shared globally affine readout for a nonconstant fixed-bank head. These are exact representation results, not pretraining, task-risk or transfer guarantees. Section 5 tests one Qwen2.5-0.5B layer.