← All AI/ML open problems

AI safety & alignment · RL & post-training · RL foundations · Control & robotics / Open question

Learning Without Catastrophic Experiments

An autonomous system may not be allowed to discover a dangerous action by trying it. Learning must use structure to distinguish safe expansion from reckless exploration.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

For finite MDPs with SS states, AA actions, rewards in [0,1][0,1] and an unknown absorbing hazard set H\mathcal H, start from a supplied safe region and backup policy. Across TT training steps require Pr⁡(∃t≤T:st∈H)≤δ\Pr(\exists t\leq T:s_t\in\mathcal H)\leq\delta. Determine the minimum interactions for value within ε\varepsilon of the best policy meeting the same safety constraint, and characterize which transition-support information or paid hazard checks make expansion beyond the initial safe region identifiable.

Assumptions and scope

  • Specify a known safe initial policy or region, a hazard predicate and structural links between observed and unobserved transitions; unsupported safe exploration can be impossible.
  • Distinguish an expected cumulative cost constraint from a high-probability trajectory-level avoidance guarantee, and include the entire training interaction history.
  • Fix reset access, safe returnability, model misspecification and what safety feedback is observed before an irreversible action; charge any simulator or reference-world queries.

What would settle it?

Matching safety–sample–performance bounds for a declared stochastic family, with an algorithm attaining safe expansion and indistinguishability examples proving where progress is impossible.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Safe exploration under GP regularity already identifies safely reachable regions. Deng and Juba give offline safe action-model learning under bounded stochastic support. The proposed extension is a sharp trajectory-risk frontier for online uncertainty and model error, not the first safe-learning algorithm. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Turchetta, Berkenkamp and Krause — Safe Exploration in Finite Markov Decision Processes with Gaussian Processes ↗partial theory · source checked 2026-11-19

    Primary search abstract checked: regularity/GP assumptions permit exploration of the safely reachable region with safety guarantees and returnability. Unconditional exploration of arbitrary unknown hazards is not claimed.

  2. Deng and Juba — Stochastic Safe Action Model Learning ↗partial theory · source checked 2026-11-19

    Publisher abstract checked: offline polynomial-time/sample learning for bounded-support stochastic action effects, permitting actions only where the dynamics model is accurate, with approximate completeness. This model-validity notion differs from a trajectory catastrophe bound.

  3. Bhamidipaty et al. — Imperfect World Models are Exploitable ↗partial theory · source checked 2026-11-19

    Abstract checked: model exploitation, limits of reward-hacking analogies and a relaxed safe planning horizon are already studied.