← All AI/ML open problems

Learning dynamics & scaling · Embeddings & representations / Open question

Learning New Features from Rewards

Frozen-feature models illuminate selection effects but cannot by themselves explain the acquisition of useful transferable features during RL.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Use horizon-H state-composition tasks on a finite S-state map alphabet, with reward only for the correct final state, and a two-layer ReLU policy whose hidden features and softmax output weights are both trained. Starting from Gaussian initialization and minibatch REINFORCE, derive a finite-width approximation valid over a growing training horizon. Quantify when learned features improve held-out map-sequence success over training only the output weights, charging both algorithms the same rollout and update budget.

Assumptions and scope

  • Specify the network parameterization, width scaling, initialization, finite-batch update, reward normalization and regularization.
  • Expose dependence on rare-success probabilities and the time interval of validity; a fixed finite-time infinite-width limit alone does not settle growing-horizon training.
  • Compare models under the same task information and cost convention, and identify the learned representation responsible for any separation.

What would settle it?

A concrete model with a quantitative finite-width/dynamical error bound and a demonstrated feature-learning advantage or obstruction, including the regime where the approximation fails.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Feature-learning infinite-width limits are established in supervised settings, and two-stage frozen-feature RL already has theory. The finite-width sparse-RL joint regime is the proposed extension. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Yang and Hu — Feature Learning in Infinite-Width Neural Networks ↗partial theory · source checked 2026-11-19

    Abstract checked: feature-learning infinite-width parameterizations and tensor-program limits are existing results, not an open existence question.

  2. Cheng et al. — How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws ↗partial theory · source checked 2026-11-19

    Full-text model and theorem statements checked: two-stage linear softmax policies with frozen features, realizability/orthogonality assumptions and gradient-flow limits. Coverage can change without new strategy support.

  3. Barzilai, Wolf and Basri — When Is Compositional Reasoning Learnable from Verifiable Rewards? ↗partial theory · source checked 2026-11-19

    Primary abstract checked: task-advantage-ratio characterization, positive learnability cases and suboptimal-composition obstructions. The remaining target must go beyond this specified autoregressive model.

  4. Shen et al. — Understanding Reasoning from Pretraining to Post-Training ↗empirical motivation · source checked 2026-11-19

    Version-three abstract checked: controlled chess pretraining/RL and a smaller math-language replication. Do not treat a fitted dependence on pretraining as a universal compute optimum.