← All AI/ML open problems

Learning dynamics & scaling / Open question

The Laws of Reinforcement Learning

A useful scaling theory should explain both when extrapolation works and when practical optimization departs from the idealized dynamics.

OpenEditorially scopedSpecified modelCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Starting from a two-stage strategy-selection/execution softmax model, derive finite-batch, discrete-update bounds identifying a regime where reward as a function of charged compute is approximated by a log-linear or log-sigmoid curve. Quantify the approximation error and its breakdown as feature interference, clipping, sampling noise or optimizer state increases.

Assumptions and scope

  • Use a fixed finite strategy/answer space and explicit feature geometry; relax realizability only with a quantified misspecification term.
  • Specify the update rule and the mapping from updates/rollouts to compute. Do not identify gradient-flow time with GPU-hours without accounting for cost.
  • The approximation is on a declared finite compute interval. A fitted ceiling is not evidence of a universal capability bound.

What would settle it?

A finite-compute approximation theorem with explicit error terms and valid parameter regimes, or a controlled counterexample showing which condition is indispensable. Reproducing a fitted sigmoid is not sufficient.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Cheng et al. already derive approximation results in a frozen-feature gradient-flow model; ScaleRL establishes empirical predictability in measured recipes. The extension concerns finite-batch practical dynamics and identifiable failure regimes. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Cheng et al. — How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws ↗partial theory · source checked 2026-11-19

    Full-text model and theorem statements checked: two-stage linear softmax policies with frozen features, realizability/orthogonality assumptions and gradient-flow limits. Coverage can change without new strategy support.

  2. Khatri et al. — The Art of Scaling Reinforcement Learning Compute for LLMs ↗empirical motivation · source checked 2026-11-19

    Abstract checked: empirical sigmoidal fits, recipe-dependent efficiency/ceilings and held-out-scale prediction; not a universal asymptotic theorem.

  3. Mroueh — RLVR: GRPO's Effective Loss, Dynamics, and Success Amplification ↗partial theory · source checked 2026-11-19

    Full-text formulation and theorem statements checked: stabilized population/no-clipping policy-space updates and KL-dependent recurrences. Finite empirical groups and shared-parameter optimization require additional analysis.

  4. Zheng et al. — An Empirical Study of Reinforcement Learning Scaling Laws ↗empirical motivation · source checked 2026-11-19

    Abstract checked: actor-critic experiments in Starpilot/Craftax, Elo-like metric and nonmonotone compute-optimal model size. This is not an LLM post-training theorem.