Alignment Without Reward Hacking
Optimizing a learned reward can exploit its blind spots. Alignment needs guarantees that survive the distribution shift created by optimization itself.
Editorially scoped
THEORIA / RESEARCH CATALOG
Find your next research direction. Explore conjectures and open questions through their models, mathematical targets and connections.
Explore with /next-steps ↗ or /attack slash commands
A source-linked collection of theoretical open problems. Explore challenges in AI safety, learning theory, transformers, world models, optimization, memory and inference. Each entry gives a precise research target, assumptions and known progress. Open status is provisional; broader programs explicitly need a framework. Sources & methodology ↗
Seed catalog · Browse the bundled collection. Search matches words; Supabase mode uses ranked full-text search.
No selection includes all.
No selection includes all.
No selection includes all.
Choose multiple options. Results match any selected value within each filter.
119 problems
Optimizing a learned reward can exploit its blind spots. Alignment needs guarantees that survive the distribution shift created by optimization itself.
Editorially scoped
A model may stop expressing information while still retaining its influence. Meaningful forgetting needs a guarantee tied to the training process and future releases.
Editorially scoped
Agents must read untrusted material without granting it the authority to redirect their actions or disclose protected information.
Editorially scoped
Matching an expert's actions need not reveal why they act that way. Reliable reward learning must describe what behavior can and cannot tell us.
Editorially scoped
If a rule can be learned from limited evidence, must every consistent dataset have a comparably small explanation? This is a precise form of the connection between simplicity and generalization.
Editorially scoped
Multi-turn certificates rely on worst-case transition and persistence properties, while practical audits observe only a finite selection of conversations. Efficiency payoff: Spend audits on safety-relevant states and transitions instead of repeatedly testing whole conversations without a coverage argument.
Editorially scoped
A model can pass extensive evaluations while retaining a rare hidden failure mode. Understanding what evidence can rule one out is a foundational problem in AI assurance.
Editorially scoped
KANs can approximate known smooth decompositions accurately; their architectural promise depends on discovering such decompositions from data. Efficiency payoff: Find a small graph of reusable univariate functions rather than paying for a wide generic model or a supplied symbolic structure.
Editorially scoped
Neural networks can represent complex interactions parsimoniously. Turning their learned features into a trustworthy scientific interaction graph requires more than a good fit.
Editorially scoped
A large pool of experts is efficient only if the router discovers their roles and sends enough learning signal to each useful expert. Efficiency payoff: Train a high-capacity model with low active computation without losing rare features through capacity overflow or early routing collapse.
Editorially scoped
Searching a graph is a simple test of whether a transformer can discover a scalable reasoning procedure.
Editorially scoped
Weight matrices are available even when training data are not. Turning their spectral signatures into reliable diagnostics could make model selection and auditing far more informative.
Editorially scoped