Learning dynamics & scaling / Open question
The Laws of Reinforcement Learning
A useful scaling theory should explain both when extrapolation works and when practical optimization departs from the idealized dynamics.
Seed catalog · Showing the bundled record.
Problem statement
Starting from a two-stage strategy-selection/execution softmax model, derive finite-batch, discrete-update bounds identifying a regime where reward as a function of charged compute is approximated by a log-linear or log-sigmoid curve. Quantify the approximation error and its breakdown as feature interference, clipping, sampling noise or optimizer state increases.
Assumptions and scope
- Use a fixed finite strategy/answer space and explicit feature geometry; relax realizability only with a quantified misspecification term.
- Specify the update rule and the mapping from updates/rollouts to compute. Do not identify gradient-flow time with GPU-hours without accounting for cost.
- The approximation is on a declared finite compute interval. A fitted ceiling is not evidence of a universal capability bound.
What would settle it?
A finite-compute approximation theorem with explicit error terms and valid parameter regimes, or a controlled counterexample showing which condition is indispensable. Reproducing a fitted sigmoid is not sufficient.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Cheng et al. already derive approximation results in a frozen-feature gradient-flow model; ScaleRL establishes empirical predictability in measured recipes. The extension concerns finite-batch practical dynamics and identifiable failure regimes. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Cheng et al. — How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws ↗partial theory · source checked 2026-11-19
Full-text model and theorem statements checked: two-stage linear softmax policies with frozen features, realizability/orthogonality assumptions and gradient-flow limits. Coverage can change without new strategy support.
- Khatri et al. — The Art of Scaling Reinforcement Learning Compute for LLMs ↗empirical motivation · source checked 2026-11-19
Abstract checked: empirical sigmoidal fits, recipe-dependent efficiency/ceilings and held-out-scale prediction; not a universal asymptotic theorem.
- Mroueh — RLVR: GRPO's Effective Loss, Dynamics, and Success Amplification ↗partial theory · source checked 2026-11-19
Full-text formulation and theorem statements checked: stabilized population/no-clipping policy-space updates and KL-dependent recurrences. Finite empirical groups and shared-parameter optimization require additional analysis.
- Zheng et al. — An Empirical Study of Reinforcement Learning Scaling Laws ↗empirical motivation · source checked 2026-11-19
Abstract checked: actor-critic experiments in Starpilot/Craftax, Elo-like metric and nonmonotone compute-optimal model size. This is not an LLM post-training theorem.