← All AI/ML open problems

RL & post-training / Open question

Stable Learning from Stale Experience

Reusing rollouts can save generation compute, but stale statistics may change both estimator bias and policy drift.

OpenEditorially scopedSpecified modelCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

For bounded-feature autoregressive softmax policies trained from a replay mixture of earlier policies, characterize the interaction of empirical group normalization, importance weighting, clipping and multiple update epochs. Derive a finite-sample improvement or stationarity guarantee with explicit dependence on policy lag and likelihood-ratio tails, and identify regimes where practical clipping fails to control drift.

Assumptions and scope

  • Freeze the exact surrogate, sampling mixture, optimizer and normalization convention before analysis; compare against a stated true-reward or regularized objective.
  • Use finite horizon and overlap conditions that are checked or explicitly parameterized, rather than assuming importance weights remain uniformly small for free.
  • Account for reuse of the same samples and the dependence between estimated advantages and the likelihood-ratio update.

What would settle it?

A sharp stability/improvement region and finite-sample bounds, accompanied by counterexamples outside it or a correction with a proved guarantee. Monotonicity of a population surrogate alone is insufficient.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

On/off-policy GRPO improvement results and population recurrences already exist. The scoped extension couples stale empirical statistics, clipping and repeated shared-parameter updates. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Mroueh et al. — Revisiting GRPO: On-Policy and Off-Policy Training ↗partial theory · source checked 2026-11-19

    Undermind abstract checked: reward-improvement results motivate off-policy clipped objectives. The complete theorem hypotheses and practical multi-epoch extensions were not independently audited.

  2. Mroueh — RLVR: GRPO's Effective Loss, Dynamics, and Success Amplification ↗partial theory · source checked 2026-11-19

    Full-text formulation and theorem statements checked: stabilized population/no-clipping policy-space updates and KL-dependent recurrences. Finite empirical groups and shared-parameter optimization require additional analysis.

  3. Khatri et al. — The Art of Scaling Reinforcement Learning Compute for LLMs ↗empirical motivation · source checked 2026-11-19

    Abstract checked: empirical sigmoidal fits, recipe-dependent efficiency/ceilings and held-out-scale prediction; not a universal asymptotic theorem.