← All AI/ML open problems

Transfer & distribution shift · Verification & proof search · AI safety & alignment / Open question

Learning in Synthetic Worlds

A synthetic environment can execute perfectly while rewarding behavior that exploits its mismatch with the intended world.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Let a generator adaptively choose horizon-H MDPs on known finite state/action sets, with rewards in [0,1] and transition tables in an r-parameter Lipschitz class; the unknown reference MDP belongs to that class. Given reference trajectories and a reset to its initial distribution, determine the audit budget needed to certify the reference-value improvement of policies trained on generated MDPs. The certificate must remain valid as both the generator and policy are selected from earlier audit results.

Assumptions and scope

  • Declare a generator/model class, reference-world access and horizon; exact execution of generated code alone is not a fidelity certificate.
  • State the discrepancy controlling value transfer and the coverage required to estimate it on policy-relevant states.
  • Charge reference-world audits and include adaptive selection of environments after earlier audit results.

What would settle it?

An audit-and-repair procedure with finite-sample transfer bounds and necessary audit/coverage lower bounds for the declared generator class. A fixed-model simulation bound is a baseline, not the full adaptive guarantee.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Empirical synthesis discounts/collapse thresholds and theoretical model-exploitation limits already exist. The targeted extension is adaptive fidelity certification during joint environment and policy generation. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. He et al. — How Data Scales in Agentic Reinforcement Learning ↗empirical motivation · source checked 2026-11-19

    Full-text Sections 3–6 checked: distinguish worlds, environment structures, tasks, trajectories and reward density. Exponents and synthesis discounts are empirical, regime-specific fits; allocation assumes a stated cost model.

  2. Bhamidipaty et al. — Imperfect World Models are Exploitable ↗partial theory · source checked 2026-11-19

    Abstract checked: model exploitation, limits of reward-hacking analogies and a relaxed safe planning horizon are already studied.

  3. Zeng et al. — RLVE: Adaptive Verifiable Environments ↗empirical motivation · source checked 2026-11-19

    ICML 2026 publisher abstract checked: 400 procedurally generating environments and capability-adaptive difficulty; empirical environment-scaling evidence.