Transfer & distribution shift · RL & post-training / Open question
Learning Without Losing Capabilities
Higher current pass@1 can coexist with loss of rare strategies needed for later tasks, making the next training stage harder.
Seed catalog · Showing the bundled record.
Problem statement
Consider two binary verifiers on the same finite prompt set and horizon- answer trees, with one bounded-feature softmax policy shared across both stages. Parameterize task compatibility by the overlap of their accepting leaves under the initial policy; later-task information is available only through a charged probe budget. Characterize the frontier between first-stage reward gain, replay storage and the later rollout cost of reaching success , with a preservation rule and matching failure examples.
Assumptions and scope
- Fix a shared-policy/task family and a measurable relationship between its valid strategy sets; incompatible tasks may force a trade-off.
- Use effective support at a stated sampling budget, not literal support of a full-support softmax model.
- Charge retained examples and later-task probes. State what information about the later task is available during the first stage.
What would settle it?
A finite-budget preservation-versus-improvement frontier with an algorithm and lower-bound examples under stated task overlap, or an impossibility result when future-task information is insufficient.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Fixed-feature transfer analysis, over-sharpening mechanisms and empirical verifier-induced support reshaping are existing results. The residual question is a quantitative sequential trainability guarantee. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Cheng et al. — How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws ↗partial theory · source checked 2026-11-19
Full-text model and theorem statements checked: two-stage linear softmax policies with frozen features, realizability/orthogonality assumptions and gradient-flow limits. Coverage can change without new strategy support.
- Wei et al. — Verifier-Induced Support Reshaping in On-Policy Optimization ↗empirical motivation · source checked 2026-11-19
Abstract checked: effective rewardable support is budget-relative; cross-task endpoint accuracy does not ensure future trainability.
- Fan et al. — When Sharpening Becomes Collapse ↗related result · source checked 2026-11-19
Abstract checked: finite-batch sampling bias and semantic coupling can suppress valid alternatives; calibration and diversification are existing proposed remedies.