RMT & weight spectra · Statistical learning theory / Open question
Can Weight Spectra Predict Generalization?
Weight matrices are available even when training data are not. Turning their spectral signatures into reliable diagnostics could make model selection and auditing far more informative.
Seed catalog · Showing the bundled record.
Problem statement
For two-layer tanh networks trained on noisy rank-k Gaussian teacher regression in proportional width/sample dimensions, suppose an auditor sees weight singular values/vectors and may buy fresh labelled probes. Determine which spectral and teacher-alignment summaries suffice for a calibrated population-risk bound across adaptively selected checkpoints, and the probe complexity needed to estimate them. A spectrum-only bound that ignores readout orientation or task labels is not the target.
Assumptions and scope
- Fix the task family, training procedure and metric normalization; the same weight spectrum can correspond to different functions or different risks under changed labels.
- Account for layer rescaling, neuron permutations and singular-vector orientation. Specify what task information is available and charge any labelled calibration data.
- Require finite-sample validity over an explicit model-selection procedure, not only correlation across a hand-selected collection of pretrained models.
What would settle it?
A calibrated risk bound or predictive-sufficiency theorem for a nontrivial restricted family, with matching ambiguity examples showing which alignment information is indispensable.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
SETOL and heavy-tailed mechanistic-universality models already connect spectral metrics to observed performance; short-step RMT analyses connect outliers to learned directions. Pure spectrum-only distribution-free prediction is not plausible, and a simple isospectral counterexample is only a baseline. The target is a quantitative task-aware diagnostic. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Martin and Hinrichs — SETOL: A Semi-Empirical Theory of (Deep) Learning ↗empirical motivation · source checked 2026-11-19
Primary abstract checked: semi-empirical spectral quality metrics and tests on MLPs/pretrained models. Spectral correlations with accuracy are not distribution-free risk certificates.
- Hodgkinson, Wang and Mahoney — Models of Heavy-Tailed Mechanistic Universality ↗related result · source checked 2026-11-19
Primary abstract checked: a proposed high-temperature Marchenko–Pastur ensemble explains heavy-tail mechanisms from correlation, temperature and eigenvector entropy. Deriving this ensemble from actual neural training is additional work.
- Moniri and Hassani — Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent ↗partial theory · source checked 2026-11-19
Primary abstract checked: two-step spiked-matrix/outlier characterization and a batch-reuse effect in proportional dimensions. Longer correlated training trajectories are a separate target.