← All AI/ML open problems

RMT & weight spectra · Learning dynamics & scaling / Open question

The Random Matrix Theory of Learning

Neural training transforms a random matrix into a structured representation. A dynamical spectral theory could explain how useful directions separate from noise.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Train a two-layer tanh network on nn Gaussian inputs labelled by a rank-kk tanh teacher plus Gaussian noise, with Gaussian initialization, both layers updated and fixed limits of width/dd and n/dn/d. Derive the empirical spectral law of the normalized first-layer Gram matrix WtWt⊤W_tW_t^\top over a growing number of minibatch SGD updates on that reused dataset. Track bulk, outliers and alignment with the teacher span; determine the training-time range of validity and whether any claimed spectral tail survives the dimensional limit.

Assumptions and scope

  • Fix width/input/sample ratios, initialization, activation, target distribution, learning-rate schedule and batch reuse. Entrywise heavy tails and heavy-tailed spectral density are distinct properties.
  • Specify the weight normalization, spectral statistic, training-time scale and convergence mode; control dependencies created by training instead of treating learned weights as independent entries.
  • Distinguish limiting bulk laws from edge fluctuations, outliers and task alignment. A bounded-range power-law fit is not proof of an asymptotic tail.

What would settle it?

A controlled nonlinear multi-step spectral law with explicit finite-size/time errors and a predicted feature-emergence regime, or a counterexample delimiting a proposed universality class.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Heavy-tailed conjugate-kernel laws, two-step spiked-weight analysis and solvable linear BBP-style learning dynamics already exist. Heavy-tailed ensemble models propose mechanisms. The residual target derives correlated nonlinear long-time spectra from the actual training process. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Guionnet and Piccolo — Global Law of Conjugate Kernel Random Matrices with Heavy-Tailed Weights ↗partial theory · source checked 2026-11-19

    Primary abstract checked: limiting spectra for a two-layer conjugate kernel with independent heavy-tailed weights and light-tailed inputs. This is not a law for arbitrary data-dependent trained weights.

  2. Moniri and Hassani — Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent ↗partial theory · source checked 2026-11-19

    Primary abstract checked: two-step spiked-matrix/outlier characterization and a batch-reuse effect in proportional dimensions. Longer correlated training trajectories are a separate target.

  3. Park et al. — Spectral Phase Transitions and Trainability in Neural Network Learning Dynamics ↗partial theory · source checked 2026-11-19

    Primary abstract checked: a solvable linear teacher–student spectral evolution, BBP-style transitions and nonlinear/stochastic extensions with numerical evidence. A general finite-time nonlinear spectral law is not established by this check.

  4. Hodgkinson, Wang and Mahoney — Models of Heavy-Tailed Mechanistic Universality ↗related result · source checked 2026-11-19

    Primary abstract checked: a proposed high-temperature Marchenko–Pastur ensemble explains heavy-tail mechanisms from correlation, temperature and eigenvector entropy. Deriving this ensemble from actual neural training is additional work.