Learning dynamics & scaling · Embeddings & representations / Open question
Learning New Features from Rewards
Frozen-feature models illuminate selection effects but cannot by themselves explain the acquisition of useful transferable features during RL.
Seed catalog · Showing the bundled record.
Problem statement
Use horizon-H state-composition tasks on a finite S-state map alphabet, with reward only for the correct final state, and a two-layer ReLU policy whose hidden features and softmax output weights are both trained. Starting from Gaussian initialization and minibatch REINFORCE, derive a finite-width approximation valid over a growing training horizon. Quantify when learned features improve held-out map-sequence success over training only the output weights, charging both algorithms the same rollout and update budget.
Assumptions and scope
- Specify the network parameterization, width scaling, initialization, finite-batch update, reward normalization and regularization.
- Expose dependence on rare-success probabilities and the time interval of validity; a fixed finite-time infinite-width limit alone does not settle growing-horizon training.
- Compare models under the same task information and cost convention, and identify the learned representation responsible for any separation.
What would settle it?
A concrete model with a quantitative finite-width/dynamical error bound and a demonstrated feature-learning advantage or obstruction, including the regime where the approximation fails.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Feature-learning infinite-width limits are established in supervised settings, and two-stage frozen-feature RL already has theory. The finite-width sparse-RL joint regime is the proposed extension. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Yang and Hu — Feature Learning in Infinite-Width Neural Networks ↗partial theory · source checked 2026-11-19
Abstract checked: feature-learning infinite-width parameterizations and tensor-program limits are existing results, not an open existence question.
- Cheng et al. — How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws ↗partial theory · source checked 2026-11-19
Full-text model and theorem statements checked: two-stage linear softmax policies with frozen features, realizability/orthogonality assumptions and gradient-flow limits. Coverage can change without new strategy support.
- Barzilai, Wolf and Basri — When Is Compositional Reasoning Learnable from Verifiable Rewards? ↗partial theory · source checked 2026-11-19
Primary abstract checked: task-advantage-ratio characterization, positive learnability cases and suboptimal-composition obstructions. The remaining target must go beyond this specified autoregressive model.
- Shen et al. — Understanding Reasoning from Pretraining to Post-Training ↗empirical motivation · source checked 2026-11-19
Version-three abstract checked: controlled chess pretraining/RL and a smaller math-language replication. Do not treat a fitted dependence on pretraining as a universal compute optimum.