Transformer theory · Embeddings & representations · Statistical learning theory / Open question
From Memorization to In-Context Learning
A foundation model must learn something reusable across tasks. Counting tokens alone may hide whether its training experience contains enough distinct tasks.
Seed catalog · Showing the bundled record.
Problem statement
Use the shared-feature regression family , with Gaussian inputs, fixed unknown orthonormal features and Gaussian label noise. Draw coefficient vectors uniformly from and reuse only these tasks during pretraining. For a finite-width softmax transformer trained on example/query prompts, determine how , examples per context and training compute control risk on a fresh coefficient vector, and whether a sharp memorization-to-transfer crossover exists.
Assumptions and scope
- Specify latent task dimension, task distribution and how independent tasks differ from repeated examples of one task.
- Fix the nonlinear attention architecture, data reuse and training procedure; require held-out task risk rather than training-task loss as the criterion.
- Define a quantitative memorization-versus-transfer diagnostic and a scaling regime. A sharp transition need not exist in every family.
What would settle it?
Matching task/sample requirements or a rigorous finite-size transition window for a gradient-trained nonlinear model, with cases distinguishing true diversity from duplicated tasks.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Wu et al. bound linear-attention task complexity; Lu et al. give an asymptotic theory; Kim et al. prove nonparametric ICL minimax bounds with linear attention and sufficient training. The target adds nonlinear optimization and finite-size crossover, rather than claiming task-complexity theory is absent. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Wu et al. — How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? ↗partial theory · source checked 2026-11-19
Abstract checked: task-complexity bound for single-layer linear attention; near Bayes-optimal ridge regression on unseen tasks.
- Lu et al. — Asymptotic Theory of In-Context Learning by Linear Attention ↗partial theory · source checked 2026-11-19
Abstract checked: sharp asymptotic characterization of ICL sample complexity and task-diversity requirements for linear attention.
- Kim, Nakamaki and Suzuki — Transformers are Minimax Optimal Nonparametric In-Context Learners ↗partial theory · source checked 2026-11-19
Abstract checked: approximation/generalization bounds for a deep network plus one linear attention layer, with lower bounds in the number of tasks; assumes sufficient training rather than analyzing optimization.
- Kwon et al. — Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspective ↗partial theory · source checked 2026-11-19
Abstract checked: linear-attention conditions for out-of-distribution ICL under subspace shifts.