AI safety & alignment · Verification & proof search / Open question
Controlling an Untrusted AI
A deployment protocol should remain reliable even when helpful behavior during evaluation does not predict behavior later.
Seed catalog · Showing the bundled record.
Problem statement
In a horizon-H finite extensive-form game, an untrusted agent proposes useful or harmful actions, a monitor sees one of m noisy signals, and a paid trusted audit reveals the action type before execution. Signal laws belong to a known finite-dimensional class but must be estimated from n evaluation episodes. Determine the minimax usefulness under a bound delta on any undetected harmful execution, allowing history-dependent adversaries and charging both evaluation and deployment audits.
Assumptions and scope
- Specify the game, unsafe-action predicate, monitor observations, trusted-audit accuracy and fallback policy; an unrestricted adversary with indistinguishable harmful actions may make useful control impossible.
- State what transfers from evaluation to deployment and which history the adversary sees, especially whether private audit randomization is hidden.
- Measure horizon-wide failure probability separately from mean per-task success; charge audits, monitoring, editing and fallback costs.
What would settle it?
A finite-data robust control protocol with a quantified safety–usefulness–audit frontier, together with lower bounds exposing necessary observability or transfer assumptions.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Greenblatt et al. evaluate control protocols under intentional subversion; Griffin et al. formalize control games and synthesize Pareto-optimal protocols; diffuse-control work studies distributed sabotage empirically. The residual target is finite-data uncertainty and adaptive long-horizon risk, not a first formal model of control. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Greenblatt et al. — AI Control: Improving Safety Despite Intentional Subversion ↗empirical motivation · source checked 2026-11-19
Primary abstract checked: trusted editing and untrusted monitoring improve measured safety/usefulness in adversarial programming-assistant evaluations. Guarantees against unrestricted adaptive adversaries are not established by the experiments.
- Griffin et al. — Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols ↗partial theory · source checked 2026-11-19
Primary search abstract checked: multi-objective partially observable stochastic games and reductions synthesize Pareto-optimal control protocols under declared models. A formal framework for AI control already exists.
- Terekhov et al. — Diffuse AI Control on Fuzzy Tasks ↗empirical motivation · source checked 2026-11-19
Primary search abstract checked: an adversarial-game evaluation of distributed sabotage in research proposals, using a stronger-model proxy scorer. Resistance to the tested red team is not a worst-case safety proof.