Alignment Without Reward Hacking
Optimizing a learned reward can exploit its blind spots. Alignment needs guarantees that survive the distribution shift created by optimization itself.
Editorially scoped
THEORIA / RESEARCH CATALOG
Find your next research direction. Explore conjectures and open questions through their models, mathematical targets and connections.
Explore with /next-steps ↗ or /attack slash commands
A source-linked collection of theoretical open problems. Explore challenges in AI safety, learning theory, transformers, world models, optimization, memory and inference. Each entry gives a precise research target, assumptions and known progress. Open status is provisional; broader programs explicitly need a framework. Sources & methodology ↗
Seed catalog · Browse the bundled collection. Search matches words; Supabase mode uses ranked full-text search.
No selection includes all.
No selection includes all.
No selection includes all.
Choose multiple options. Results match any selected value within each filter.
18 problems
Optimizing a learned reward can exploit its blind spots. Alignment needs guarantees that survive the distribution shift created by optimization itself.
Editorially scoped
A model may stop expressing information while still retaining its influence. Meaningful forgetting needs a guarantee tied to the training process and future releases.
Editorially scoped
Agents must read untrusted material without granting it the authority to redirect their actions or disclose protected information.
Editorially scoped
Matching an expert's actions need not reveal why they act that way. Reliable reward learning must describe what behavior can and cannot tell us.
Editorially scoped
Multi-turn certificates rely on worst-case transition and persistence properties, while practical audits observe only a finite selection of conversations. Efficiency payoff: Spend audits on safety-relevant states and transitions instead of repeatedly testing whole conversations without a coverage argument.
Editorially scoped
A model can pass extensive evaluations while retaining a rare hidden failure mode. Understanding what evidence can rule one out is a foundational problem in AI assurance.
Editorially scoped
A defense must protect against an entire specified perturbation set while remaining useful. Stronger detection can otherwise buy safety by rejecting most requests. Efficiency payoff: Reduce the number of extra generations or exhaustive edit checks needed for a meaningful robustness/utility certificate.
Editorially scoped
A deployment protocol should remain reliable even when helpful behavior during evaluation does not predict behavior later.
Editorially scoped
High average accuracy is not enough when mistakes concentrate on the people or situations least visible to the evaluator.
Editorially scoped
Can a learner reliably become better than the source of its supervision? The answer matters whenever strong models are trained using weaker judges.
Editorially scoped
A synthetic environment can execute perfectly while rewarding behavior that exploits its mismatch with the intended world.
Editorially scoped
An autonomous system may not be allowed to discover a dangerous action by trying it. Learning must use structure to distinguish safe expansion from reckless exploration.
Editorially scoped