Kernel learning · Learning dynamics & scaling / Open question
How Long Does the NTK Describe Classification?
The width-to-infinity and time-to-infinity limits can disagree. Regularization and soft labels can stabilize a kernel description that hard-label training eventually leaves. Efficiency payoff: Know when cheap kernel predictions of training remain trustworthy and when continued computation is changing the representation.
Seed catalog · Showing the bundled record.
Problem statement
Train from Gaussian initialization on Gaussian inputs with labels , . Use cross-entropy targets and penalty . With empirical NTK , determine the first iteration when , jointly in width, sample size and vanishing .
Assumptions and scope
- All a_j and w_j coordinates are initialized independently N(0,1). Train both layers by full-batch gradient descent with a fixed positive step h on mean cross-entropy plus the displayed penalty; h and eta in (0,1) are explicit inputs, and a positive theorem must give its admissible h range.
- T_eta is infinity if the threshold is never crossed. Give high-probability upper/lower bounds over initialization and samples, exposing any conditioning on the initial Gram eigenvalues. K_0 is the finite random kernel; approximating it by a deterministic infinite-width kernel creates a separate error.
- Relate kernel drift to the prediction discrepancy from training the initial linearized model with the same loss/penalty and steps. A norm change alone does not establish improved generalization, and hard-label separability cannot be replaced by a positive soft-label floor without disclosure.
What would settle it?
A quantitative validity/crossover law with explicit width, stepsize, sample and stabilizer dependence, recovering a nontrivial stable regime and a demonstrable breakdown regime, plus prediction-error control on the valid interval.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Finite-width classification breakdown and regularized/nondegenerate-target NTK classification already have theory. Sharp squared-loss rescaling-time bounds and finite depth/width corrections also exist. The added benchmark asks for a joint finite-width timescale as stabilization vanishes, not the existence of kernel convergence or breakdown. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Plenk et al. — The Neural Tangent Kernel for Classification ↗partial theory · source checked 2026-11-20
Primary and scholarly abstracts checked: parameter regularization or nondegenerate soft targets recover lazy classification dynamics under stated conditions. The additional target concerns quantitative finite-width crossover as those stabilizers vanish.
- Yu, Tian and Chen — Divergence of Empirical Neural Tangent Kernel in Classification Problems ↗partial theory · source checked 2026-11-20
Scholarly abstract checked: no uniform-all-time NTK approximation in the analyzed finite-width hard-label classification settings. The proposed target is a sharp validity timescale, not whether breakdown is possible.
- Boix-Adserà and Littwin — The NTK Approximation Is Valid for Longer Than You Think ↗partial theory · source checked 2026-11-20
Scholarly abstract checked: sharp model-rescaling versus training-horizon bounds for squared loss. A squared-loss rescaling theorem cannot be silently transferred to hard-label cross-entropy or identified with width scaling.
- Hanin and Nica — Finite Depth and Width Corrections to the Neural Tangent Kernel ↗partial theory · source checked 2026-11-20
Scholarly abstract checked: finite depth/width moments and first-update effects already show nontrivial joint-limit behavior. Initial-kernel concentration is not a full long-time training guarantee.