← All AI/ML open problems

Embeddings & representations · Transformer theory · Learning dynamics & scaling / Open question

Why Tiny Updates Unlock New Capabilities

Small updates can change a foundation model's behavior dramatically. Understanding when this is enough would connect representation geometry to efficient adaptation. Efficiency payoff: Allocate small adapters and loop-specific corrections instead of duplicating complete network layers.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Use Gaussian regression tasks whose labels depend on k shared ReLU features. Downstream tasks change the feature span by s new directions at principal angle at least alpha and change bounded output coefficients; the original task distribution remains the retention test. For a pretrained softmax transformer given example/query prompts, characterize the layerwise LoRA ranks needed for gradient fine-tuning to match a full-weight-update learner's risk, at equal data access, including explicit adaptation and retention errors.

Assumptions and scope

  • Specify pretraining information, downstream task geometry, architecture and exactly which weight matrices receive adapters; teacher shifts must be defined independently of the desired answer.
  • Fix factor initialization, optimizer, data budget and rank constraints. Representability by low-rank updates alone is not a trainability guarantee.
  • Compare full and low-rank updates at a stated cost or sample budget, measuring adaptation and retention on separately declared distributions.

What would settle it?

A finite-sample optimization and rank-allocation theorem with a matching insufficiency example, including a quantified adaptation-versus-retention trade-off in a declared multi-layer family.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

LoRA expressivity bounds, rank-one two-layer online-SGD convergence and stylized one-layer transformer knowledge extraction are known. The target is multi-layer rank allocation with training and retention guarantees, not the first explanation of low-rank adaptation. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Dayi and Chen — Low-Rank Fine-Tuning Lies Between Lazy Training and Feature Learning ↗partial theory · source checked 2026-11-19

    Publisher abstract checked: rank-one teacher perturbation of a two-layer base on Gaussian inputs, with an online-SGD convergence guarantee and an advantage over learning from scratch.

  2. Zeng and Lee — The Expressive Power of Low-Rank Adaptation ↗partial theory · source checked 2026-11-19

    Primary search abstract checked: rank-dependent approximation and transformer adaptation constructions. Existence of adapter weights does not prove that practical fine-tuning finds them.

  3. Xu and Chen — Provable Knowledge Acquisition and Extraction in One-Layer Transformers ↗partial theory · source checked 2026-11-19

    Primary search abstract checked: stylized pretraining/fine-tuning dynamics and relation-coverage guarantees for both full and low-rank adaptation. Multi-layer rank allocation is a separate extension.

  4. Bae et al. — Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA ↗related algorithm · source checked 2026-11-20

    Google DeepMind publication page checked: shared blocks, loop-specific low-rank adapters and depth-wise batching. Weight tying and early-exit scheduling already exist; repeated arithmetic and storage savings differ.