Embeddings & representations · Transformer theory · Learning dynamics & scaling / Open question
Why Tiny Updates Unlock New Capabilities
Small updates can change a foundation model's behavior dramatically. Understanding when this is enough would connect representation geometry to efficient adaptation. Efficiency payoff: Allocate small adapters and loop-specific corrections instead of duplicating complete network layers.
Seed catalog · Showing the bundled record.
Problem statement
Use Gaussian regression tasks whose labels depend on k shared ReLU features. Downstream tasks change the feature span by s new directions at principal angle at least alpha and change bounded output coefficients; the original task distribution remains the retention test. For a pretrained softmax transformer given example/query prompts, characterize the layerwise LoRA ranks needed for gradient fine-tuning to match a full-weight-update learner's risk, at equal data access, including explicit adaptation and retention errors.
Assumptions and scope
- Specify pretraining information, downstream task geometry, architecture and exactly which weight matrices receive adapters; teacher shifts must be defined independently of the desired answer.
- Fix factor initialization, optimizer, data budget and rank constraints. Representability by low-rank updates alone is not a trainability guarantee.
- Compare full and low-rank updates at a stated cost or sample budget, measuring adaptation and retention on separately declared distributions.
What would settle it?
A finite-sample optimization and rank-allocation theorem with a matching insufficiency example, including a quantified adaptation-versus-retention trade-off in a declared multi-layer family.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
LoRA expressivity bounds, rank-one two-layer online-SGD convergence and stylized one-layer transformer knowledge extraction are known. The target is multi-layer rank allocation with training and retention guarantees, not the first explanation of low-rank adaptation. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Dayi and Chen — Low-Rank Fine-Tuning Lies Between Lazy Training and Feature Learning ↗partial theory · source checked 2026-11-19
Publisher abstract checked: rank-one teacher perturbation of a two-layer base on Gaussian inputs, with an online-SGD convergence guarantee and an advantage over learning from scratch.
- Zeng and Lee — The Expressive Power of Low-Rank Adaptation ↗partial theory · source checked 2026-11-19
Primary search abstract checked: rank-dependent approximation and transformer adaptation constructions. Existence of adapter weights does not prove that practical fine-tuning finds them.
- Xu and Chen — Provable Knowledge Acquisition and Extraction in One-Layer Transformers ↗partial theory · source checked 2026-11-19
Primary search abstract checked: stylized pretraining/fine-tuning dynamics and relation-coverage guarantees for both full and low-rank adaptation. Multi-layer rank allocation is a separate extension.
- Bae et al. — Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA ↗related algorithm · source checked 2026-11-20
Google DeepMind publication page checked: shared blocks, loop-specific low-rank adapters and depth-wise batching. Weight tying and early-exit scheduling already exist; repeated arithmetic and storage savings differ.