← All AI/ML open problems

Kernel learning · Learning dynamics & scaling / Open question

How Long Does the NTK Describe Classification?

The width-to-infinity and time-to-infinity limits can disagree. Regularization and soft labels can stabilize a kernel description that hard-label training eventually leaves. Efficiency payoff: Know when cheap kernel predictions of training remain trustworthy and when continued computation is changing the representation.

OpenEditorially scopedSpecified modelCompiled 2026-11-20

Seed catalog · Showing the bundled record.

Problem statement

Train fθ(x)=m−1/2∑j=1majtanh⁡(wj⊤x)f_\theta(x)=m^{-1/2}\sum_{j=1}^m a_j\tanh(w_j^\top x) from Gaussian initialization on nn Gaussian inputs with labels yi=1{u⊤xi>0}y_i=\mathbf1\{u^\top x_i>0\}, ∥u∥=1\|u\|=1. Use cross-entropy targets qi=(1−ρ)yi+ρ/2q_i=(1-\rho)y_i+\rho/2 and penalty λ∥θ−θ0∥2/2\lambda\|\theta-\theta_0\|^2/2. With empirical NTK Kt(i,j)=⟨∇fθt(xi),∇fθt(xj)⟩K_t(i,j)=\langle\nabla f_{\theta_t}(x_i),\nabla f_{\theta_t}(x_j)\rangle, determine the first iteration TηT_\eta when ∥Kt−K0∥op>η∥K0∥op\|K_t-K_0\|_{\mathrm{op}}>\eta\|K_0\|_{\mathrm{op}}, jointly in width, sample size and vanishing λ,ρ\lambda,\rho.

Assumptions and scope

  • All a_j and w_j coordinates are initialized independently N(0,1). Train both layers by full-batch gradient descent with a fixed positive step h on mean cross-entropy plus the displayed penalty; h and eta in (0,1) are explicit inputs, and a positive theorem must give its admissible h range.
  • T_eta is infinity if the threshold is never crossed. Give high-probability upper/lower bounds over initialization and samples, exposing any conditioning on the initial Gram eigenvalues. K_0 is the finite random kernel; approximating it by a deterministic infinite-width kernel creates a separate error.
  • Relate kernel drift to the prediction discrepancy from training the initial linearized model with the same loss/penalty and steps. A norm change alone does not establish improved generalization, and hard-label separability cannot be replaced by a positive soft-label floor without disclosure.

What would settle it?

A quantitative validity/crossover law with explicit width, stepsize, sample and stabilizer dependence, recovering a nontrivial stable regime and a demonstrable breakdown regime, plus prediction-error control on the valid interval.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Finite-width classification breakdown and regularized/nondegenerate-target NTK classification already have theory. Sharp squared-loss rescaling-time bounds and finite depth/width corrections also exist. The added benchmark asks for a joint finite-width timescale as stabilization vanishes, not the existence of kernel convergence or breakdown. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Plenk et al. — The Neural Tangent Kernel for Classification ↗partial theory · source checked 2026-11-20

    Primary and scholarly abstracts checked: parameter regularization or nondegenerate soft targets recover lazy classification dynamics under stated conditions. The additional target concerns quantitative finite-width crossover as those stabilizers vanish.

  2. Yu, Tian and Chen — Divergence of Empirical Neural Tangent Kernel in Classification Problems ↗partial theory · source checked 2026-11-20

    Scholarly abstract checked: no uniform-all-time NTK approximation in the analyzed finite-width hard-label classification settings. The proposed target is a sharp validity timescale, not whether breakdown is possible.

  3. Boix-Adserà and Littwin — The NTK Approximation Is Valid for Longer Than You Think ↗partial theory · source checked 2026-11-20

    Scholarly abstract checked: sharp model-rescaling versus training-horizon bounds for squared loss. A squared-loss rescaling theorem cannot be silently transferred to hard-label cross-entropy or identified with width scaling.

  4. Hanin and Nica — Finite Depth and Width Corrections to the Neural Tangent Kernel ↗partial theory · source checked 2026-11-20

    Scholarly abstract checked: finite depth/width moments and first-update effects already show nontrivial joint-limit behavior. Initial-kernel concentration is not a full long-time training guarantee.