RL & post-training / Open question
Stable Learning from Stale Experience
Reusing rollouts can save generation compute, but stale statistics may change both estimator bias and policy drift.
Seed catalog · Showing the bundled record.
Problem statement
For bounded-feature autoregressive softmax policies trained from a replay mixture of earlier policies, characterize the interaction of empirical group normalization, importance weighting, clipping and multiple update epochs. Derive a finite-sample improvement or stationarity guarantee with explicit dependence on policy lag and likelihood-ratio tails, and identify regimes where practical clipping fails to control drift.
Assumptions and scope
- Freeze the exact surrogate, sampling mixture, optimizer and normalization convention before analysis; compare against a stated true-reward or regularized objective.
- Use finite horizon and overlap conditions that are checked or explicitly parameterized, rather than assuming importance weights remain uniformly small for free.
- Account for reuse of the same samples and the dependence between estimated advantages and the likelihood-ratio update.
What would settle it?
A sharp stability/improvement region and finite-sample bounds, accompanied by counterexamples outside it or a correction with a proved guarantee. Monotonicity of a population surrogate alone is insufficient.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
On/off-policy GRPO improvement results and population recurrences already exist. The scoped extension couples stale empirical statistics, clipping and repeated shared-parameter updates. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Mroueh et al. — Revisiting GRPO: On-Policy and Off-Policy Training ↗partial theory · source checked 2026-11-19
Undermind abstract checked: reward-improvement results motivate off-policy clipped objectives. The complete theorem hypotheses and practical multi-epoch extensions were not independently audited.
- Mroueh — RLVR: GRPO's Effective Loss, Dynamics, and Success Amplification ↗partial theory · source checked 2026-11-19
Full-text formulation and theorem statements checked: stabilized population/no-clipping policy-space updates and KL-dependent recurrences. Finite empirical groups and shared-parameter optimization require additional analysis.
- Khatri et al. — The Art of Scaling Reinforcement Learning Compute for LLMs ↗empirical motivation · source checked 2026-11-19
Abstract checked: empirical sigmoidal fits, recipe-dependent efficiency/ceilings and held-out-scale prediction; not a universal asymptotic theorem.