AI safety & alignment · RL & post-training · RL foundations · Control & robotics / Open question
Learning Without Catastrophic Experiments
An autonomous system may not be allowed to discover a dangerous action by trying it. Learning must use structure to distinguish safe expansion from reckless exploration.
Seed catalog · Showing the bundled record.
Problem statement
For finite MDPs with states, actions, rewards in and an unknown absorbing hazard set , start from a supplied safe region and backup policy. Across training steps require . Determine the minimum interactions for value within of the best policy meeting the same safety constraint, and characterize which transition-support information or paid hazard checks make expansion beyond the initial safe region identifiable.
Assumptions and scope
- Specify a known safe initial policy or region, a hazard predicate and structural links between observed and unobserved transitions; unsupported safe exploration can be impossible.
- Distinguish an expected cumulative cost constraint from a high-probability trajectory-level avoidance guarantee, and include the entire training interaction history.
- Fix reset access, safe returnability, model misspecification and what safety feedback is observed before an irreversible action; charge any simulator or reference-world queries.
What would settle it?
Matching safety–sample–performance bounds for a declared stochastic family, with an algorithm attaining safe expansion and indistinguishability examples proving where progress is impossible.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Safe exploration under GP regularity already identifies safely reachable regions. Deng and Juba give offline safe action-model learning under bounded stochastic support. The proposed extension is a sharp trajectory-risk frontier for online uncertainty and model error, not the first safe-learning algorithm. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Turchetta, Berkenkamp and Krause — Safe Exploration in Finite Markov Decision Processes with Gaussian Processes ↗partial theory · source checked 2026-11-19
Primary search abstract checked: regularity/GP assumptions permit exploration of the safely reachable region with safety guarantees and returnability. Unconditional exploration of arbitrary unknown hazards is not claimed.
- Deng and Juba — Stochastic Safe Action Model Learning ↗partial theory · source checked 2026-11-19
Publisher abstract checked: offline polynomial-time/sample learning for bounded-support stochastic action effects, permitting actions only where the dynamics model is accurate, with approximate completeness. This model-validity notion differs from a trajectory catastrophe bound.
- Bhamidipaty et al. — Imperfect World Models are Exploitable ↗partial theory · source checked 2026-11-19
Abstract checked: model exploitation, limits of reward-hacking analogies and a relaxed safe planning horizon are already studied.