RMT & weight spectra · Learning dynamics & scaling / Open question
The Random Matrix Theory of Learning
Neural training transforms a random matrix into a structured representation. A dynamical spectral theory could explain how useful directions separate from noise.
Seed catalog · Showing the bundled record.
Problem statement
Train a two-layer tanh network on Gaussian inputs labelled by a rank- tanh teacher plus Gaussian noise, with Gaussian initialization, both layers updated and fixed limits of width/ and . Derive the empirical spectral law of the normalized first-layer Gram matrix over a growing number of minibatch SGD updates on that reused dataset. Track bulk, outliers and alignment with the teacher span; determine the training-time range of validity and whether any claimed spectral tail survives the dimensional limit.
Assumptions and scope
- Fix width/input/sample ratios, initialization, activation, target distribution, learning-rate schedule and batch reuse. Entrywise heavy tails and heavy-tailed spectral density are distinct properties.
- Specify the weight normalization, spectral statistic, training-time scale and convergence mode; control dependencies created by training instead of treating learned weights as independent entries.
- Distinguish limiting bulk laws from edge fluctuations, outliers and task alignment. A bounded-range power-law fit is not proof of an asymptotic tail.
What would settle it?
A controlled nonlinear multi-step spectral law with explicit finite-size/time errors and a predicted feature-emergence regime, or a counterexample delimiting a proposed universality class.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Heavy-tailed conjugate-kernel laws, two-step spiked-weight analysis and solvable linear BBP-style learning dynamics already exist. Heavy-tailed ensemble models propose mechanisms. The residual target derives correlated nonlinear long-time spectra from the actual training process. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Guionnet and Piccolo — Global Law of Conjugate Kernel Random Matrices with Heavy-Tailed Weights ↗partial theory · source checked 2026-11-19
Primary abstract checked: limiting spectra for a two-layer conjugate kernel with independent heavy-tailed weights and light-tailed inputs. This is not a law for arbitrary data-dependent trained weights.
- Moniri and Hassani — Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent ↗partial theory · source checked 2026-11-19
Primary abstract checked: two-step spiked-matrix/outlier characterization and a batch-reuse effect in proportional dimensions. Longer correlated training trajectories are a separate target.
- Park et al. — Spectral Phase Transitions and Trainability in Neural Network Learning Dynamics ↗partial theory · source checked 2026-11-19
Primary abstract checked: a solvable linear teacher–student spectral evolution, BBP-style transitions and nonlinear/stochastic extensions with numerical evidence. A general finite-time nonlinear spectral law is not established by this check.
- Hodgkinson, Wang and Mahoney — Models of Heavy-Tailed Mechanistic Universality ↗related result · source checked 2026-11-19
Primary abstract checked: a proposed high-temperature Marchenko–Pastur ensemble explains heavy-tail mechanisms from correlation, temperature and eigenvector entropy. Deriving this ensemble from actual neural training is additional work.