← All AI/ML open problems

Transfer & distribution shift · RL & post-training / Open question

Learning Without Losing Capabilities

Higher current pass@1 can coexist with loss of rare strategies needed for later tasks, making the next training stage harder.

OpenEditorially scopedSpecified modelCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Consider two binary verifiers on the same finite prompt set and horizon-HH answer trees, with one bounded-feature softmax policy shared across both stages. Parameterize task compatibility by the overlap of their accepting leaves under the initial policy; later-task information is available only through a charged probe budget. Characterize the frontier between first-stage reward gain, replay storage and the later rollout cost of reaching success 1−ε1-\varepsilon, with a preservation rule and matching failure examples.

Assumptions and scope

  • Fix a shared-policy/task family and a measurable relationship between its valid strategy sets; incompatible tasks may force a trade-off.
  • Use effective support at a stated sampling budget, not literal support of a full-support softmax model.
  • Charge retained examples and later-task probes. State what information about the later task is available during the first stage.

What would settle it?

A finite-budget preservation-versus-improvement frontier with an algorithm and lower-bound examples under stated task overlap, or an impossibility result when future-task information is insufficient.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Fixed-feature transfer analysis, over-sharpening mechanisms and empirical verifier-induced support reshaping are existing results. The residual question is a quantitative sequential trainability guarantee. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Cheng et al. — How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws ↗partial theory · source checked 2026-11-19

    Full-text model and theorem statements checked: two-stage linear softmax policies with frozen features, realizability/orthogonality assumptions and gradient-flow limits. Coverage can change without new strategy support.

  2. Wei et al. — Verifier-Induced Support Reshaping in On-Policy Optimization ↗empirical motivation · source checked 2026-11-19

    Abstract checked: effective rewardable support is budget-relative; cross-task endpoint accuracy does not ensure future trainability.

  3. Fan et al. — When Sharpening Becomes Collapse ↗related result · source checked 2026-11-19

    Abstract checked: finite-batch sampling bias and semantic coupling can suppress valid alternatives; calibration and diversification are existing proposed remedies.