Learning dynamics & scaling · Data-efficient learning / Open question
Learning Without Losing the Ability to Learn
Saturation and redundant neurons can trap training dynamics, so a compact network may cease adapting even when new data are informative. Efficiency payoff: Preserve useful features while rejuvenating only the capacity needed for change, reducing full retraining and replay requirements.
Seed catalog · Showing the bundled record.
Problem statement
Observe and , where , , and independent noise has variance at most . The teacher path is fixed in advance with variation . For a width- student using one stochastic-gradient update per example and at most neuron resets, determine achievable tracking regret as a function of .
Assumptions and scope
- Predict before seeing y_t; independent inputs/noise and an oblivious teacher path prevent anticipatory leakage. Teacher unit labels are fixed when defining variation. A useful subfamily may impose a quantified feature-separation condition, but must report its necessity and constants.
- Fix initialization, learning-rate/projection rules, which units may be reset and their reset distribution before a positive theorem. Count all model/optimizer state in O(md) storage and prohibit replay in the base comparison; an added replay buffer must be charged.
- Require a vanishing average tracking error in a specified nontrivial regime such as V_T=o(T), with static-teacher estimation cost and reset cost explicit. Compare to retraining and to the same-width update without resets; do not infer learning from a covariance-rank statistic alone.
What would settle it?
A finite-width update/reset scheme with a quantitative tracking bound and a failure or lower-bound regime for too little plasticity. Connect the guarantee to escaping or avoiding the identified invariant/trapping sets.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Stable-manifold explanations of plasticity loss and utility-weighted perturbation methods already exist. The proposed teacher-path and compute constraints sharpen the empirical agenda into a research benchmark; the optimizer/reset design remains to be fixed and the exact frontier is not certified novel. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Joudaki et al. — Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity ↗partial theory · source checked 2026-11-20
Primary abstract checked: stable trapping manifolds from saturated/cloned units explain a simplicity–adaptability tension. Mathematical plasticity-loss mechanisms already exist; finite-width tracking under drifting tasks is a separate target.
- Elsayed and Mahmood — Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning ↗related algorithm · source checked 2026-11-20
Primary search abstract checked in the lab review: utility-weighted perturbations protect useful units and rejuvenate others. Empirical gains do not establish the drifting-teacher regret frontier below.
- Ineffable Intelligence — mission, beliefs and A Note from Dave ↗research agenda · source checked 2026-11-20
Primary site checked: experience-driven, continual learning without reliance on human data is the stated agenda. The site supplies no corresponding theorem; rewards, environment access and architectures are still inductive assumptions.