Learning dynamics & scaling · Statistical learning theory / Open question
Scaling Laws from First Principles
Scaling laws guide enormous computational investments. A predictive theory should derive their range of validity and failures from the learning problem itself.
Seed catalog · Showing the bundled record.
Problem statement
Let inputs be independent Gaussian leaf variables of a depth-h tree; each internal node applies a normalized bounded-degree polynomial to a fixed number of children, and labels are a noisy root value. For finite-width ReLU networks trained by minibatch SGD with data reuse, derive prediction-risk dependence on tree depth/branching, samples, width and update compute. Determine whether a power-law regime exists and how interactions and feature-learning bottlenecks change its exponents and compute-optimal allocation.
Assumptions and scope
- Specify task hierarchy, feature dependence and data spectrum or correlation law independently of observed learning curves.
- Fix architecture, parameterization, initialization, optimizer and batch/data-reuse protocol, with an explicit FLOP cost model.
- State the limit order and finite-size validity range; different resource regimes may have different exponents or no power law.
What would settle it?
A controlled risk approximation with proved exponents and phase boundaries for an interacting-feature family, including a derived compute optimum and a quantitative comparison to a frozen-feature baseline.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Solvable random-feature laws, compute-optimal phase diagrams, beyond-kernel scaling and rigorous shallow online-SGD exponents already exist. Cagnetta et al. predict data-limited language exponents. The target is a joint finite-compute theory for dependent hierarchical features, not the first derivation of any exponent. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Paquette et al. — 4+3 Phases of Compute-Optimal Neural Scaling Laws ↗partial theory · source checked 2026-11-19
Abstract checked: proven compute-optimal exponents for a solvable linear power-law model trained by one-pass SGD.
- Bordelon, Atanasov and Pehlevan — A Dynamical Model of Neural Scaling Laws ↗partial theory · source checked 2026-11-19
Abstract checked: random-feature gradient-descent model explaining distinct time and size exponents and asymmetric compute-optimal scaling.
- Bordelon, Atanasov and Pehlevan — How Feature Learning Can Improve Neural Scaling Laws ↗partial theory · source checked 2026-11-19
Abstract checked: solvable beyond-kernel model; feature learning nearly doubles the time exponent only for tasks outside the initial NTK RKHS.
- Ren et al. — Emergence and Scaling Laws in SGD Learning of Shallow Neural Networks ↗partial theory · source checked 2026-11-19
Abstract checked: rigorous online-SGD analysis of extensive-width additive single-index targets; scaling exponents in samples, steps and parameters from many emergent transitions.
- Nam et al. — An Exactly Solvable Model for Emergence and Scaling Laws in the Multitask Sparse Parity Problem ↗partial theory · source checked 2026-11-19
Abstract checked: a solvable skill-basis model compared with two-layer network simulations of multitask sparse parity.
- Cagnetta et al. — Deriving Neural Scaling Laws from the Statistics of Natural Language ↗partial theory · source checked 2026-11-19
Abstract checked: data-limited scaling exponents predicted from token-correlation and conditional-entropy decay, with empirical agreement on TinyStories and WikiText. A joint compute-limited training theorem is a separate target.