← All AI/ML open problems

Kernel learning · Embeddings & representations · Statistical learning theory / Open question

Learning the Right Kernel

Kernel methods need not keep their representation fixed. Understanding how they learn a useful similarity geometry could bridge classical statistics and neural feature learning.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Let x∼N(0,Id)x\sim N(0,I_d) and y=g(U⊤x)+ξy=g(U^\top x)+\xi, where U⊤U=IkU^\top U=I_k, gg is a degree-at-most-qq polynomial with bounded coefficients and nondegenerate gradient covariance, and ξ\xi is Gaussian noise. Start with an isotropic Gaussian kernel, alternate ridge fitting with damped, trace-normalized empirical gradient-outer-product metric updates, and reuse the same sample. Prove multi-round feature-subspace and prediction-risk bounds relative to the initial kernel, or identify a failure regime caused by symmetry, noise or unstable reuse.

Assumptions and scope

  • Fix a positive-definite kernel family, ridge strength, average-gradient-outer-product update, normalization and damping; the learned kernel must be well defined on fresh points.
  • Specify input/target structure, noise and the sample-reuse protocol. An accurate population gradient outer product cannot be assumed for free.
  • Measure both representation recovery and population prediction risk, and charge kernel solves or their approximations. One successful feature update does not guarantee the full recursive trajectory.

What would settle it?

A controlled nonlinear multi-round learning theorem with sample/iteration complexity and a strict matched fixed-kernel advantage, plus failure regimes for symmetry, noise or unstable reuse.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Linear RFM trajectories already have finite-sample recovery theory; AGOP already recovers multi-index subspaces before accurate prediction; algebraic-task symmetry effects have been studied empirically. The target is a joint nonlinear recursive optimization-and-generalization result. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Cheng et al. — Exact Dynamics and Finite-Sample Trajectory Recovery of Linear Recursive Feature Machines ↗partial theory · source checked 2026-11-19

    Primary abstract checked: regularized noisy multi-output linear RFM trajectories remain close to an infinite-data reference at every iteration under stated sub-Gaussian assumptions.

  2. Zhu et al. — Average Gradient Outer Product in Kernel Regression Provably Recovers the Central Subspace for Multi-Index Models ↗partial theory · source checked 2026-11-19

    Primary abstract checked: AGOP subspace recovery can require fewer samples than accurate fixed-kernel prediction when low-degree components expose all relevant directions. This is an established representation/prediction separation.

  3. Bernal, Mallinar and Belkin — Breaking Data Symmetry Is Needed for Generalization in Feature Learning Kernels ↗empirical motivation · source checked 2026-11-19

    Primary abstract checked: empirical RFM grokking and symmetry-breaking mechanisms on algebraic tasks. The title's necessity claim is not imported as a universal theorem.