← All AI/ML open problems

Transformer theory · Embeddings & representations · Statistical learning theory / Open question

From Memorization to In-Context Learning

A foundation model must learn something reusable across tasks. Counting tokens alone may hide whether its training experience contains enough distinct tasks.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Use the shared-feature regression family fa(x)=a1ReLU⁡(w1⊤x)+a2ReLU⁡(w2⊤x)f_a(x)=a_1\operatorname{ReLU}(w_1^\top x)+a_2\operatorname{ReLU}(w_2^\top x), with Gaussian inputs, fixed unknown orthonormal features and Gaussian label noise. Draw MM coefficient vectors uniformly from [−1,1]2[-1,1]^2 and reuse only these tasks during pretraining. For a finite-width softmax transformer trained on example/query prompts, determine how MM, examples per context and training compute control risk on a fresh coefficient vector, and whether a sharp memorization-to-transfer crossover exists.

Assumptions and scope

  • Specify latent task dimension, task distribution and how independent tasks differ from repeated examples of one task.
  • Fix the nonlinear attention architecture, data reuse and training procedure; require held-out task risk rather than training-task loss as the criterion.
  • Define a quantitative memorization-versus-transfer diagnostic and a scaling regime. A sharp transition need not exist in every family.

What would settle it?

Matching task/sample requirements or a rigorous finite-size transition window for a gradient-trained nonlinear model, with cases distinguishing true diversity from duplicated tasks.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Wu et al. bound linear-attention task complexity; Lu et al. give an asymptotic theory; Kim et al. prove nonparametric ICL minimax bounds with linear attention and sufficient training. The target adds nonlinear optimization and finite-size crossover, rather than claiming task-complexity theory is absent. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Wu et al. — How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? ↗partial theory · source checked 2026-11-19

    Abstract checked: task-complexity bound for single-layer linear attention; near Bayes-optimal ridge regression on unseen tasks.

  2. Lu et al. — Asymptotic Theory of In-Context Learning by Linear Attention ↗partial theory · source checked 2026-11-19

    Abstract checked: sharp asymptotic characterization of ICL sample complexity and task-diversity requirements for linear attention.

  3. Kim, Nakamaki and Suzuki — Transformers are Minimax Optimal Nonparametric In-Context Learners ↗partial theory · source checked 2026-11-19

    Abstract checked: approximation/generalization bounds for a deep network plus one linear attention layer, with lower bounds in the number of tasks; assumes sufficient training rather than analyzing optimization.

  4. Kwon et al. — Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspective ↗partial theory · source checked 2026-11-19

    Abstract checked: linear-attention conditions for out-of-distribution ICL under subspace shifts.