THEORIA / RESEARCH CATALOG

Open problems
in foundations of AI/ML.

Find your next research direction. Explore conjectures and open questions through their models, mathematical targets and connections.

Theory & computationMathematical models & guarantees
119open problems

Explore with /next-steps ↗ or /attack slash commands

A source-linked collection of theoretical open problems. Explore challenges in AI safety, learning theory, transformers, world models, optimization, memory and inference. Each entry gives a precise research target, assumptions and known progress. Open status is provisional; broader programs explicitly need a framework. Sources & methodology ↗

Seed catalog · Browse the bundled collection. Search matches words; Supabase mode uses ranked full-text search.

START EXPLORING
SubjectAI safety & alignment
TypeAll types
ReviewAll entries

Choose multiple options. Results match any selected value within each filter.

18 problems

AI safety & alignment · RL & post-trainingOpen question

Alignment Without Reward Hacking

Optimizing a learned reward can exploit its blind spots. Alignment needs guarantees that survive the distribution shift created by optimization itself.

Editorially scoped

4 sources · Framework needs definitionRead
AI safety & alignment · Statistical learning theoryOpen question

Can a Model Truly Forget?

A model may stop expressing information while still retaining its influence. Meaningful forgetting needs a guarantee tied to the training process and future releases.

Editorially scoped

3 sources · Framework needs definitionRead
AI safety & alignment · Verification & proof searchOpen question

Can Agents Resist Prompt Injection?

Agents must read untrusted material without granting it the authority to redirect their actions or disclose protected information.

Editorially scoped

2 sources · Framework needs definitionRead
Inverse RL & imitation · Statistical inference · AI safety & alignmentOpen question

Can Behavior Reveal the Reward?

Matching an expert's actions need not reveal why they act that way. Reliable reward learning must describe what behavior can and cannot tell us.

Editorially scoped

3 sources · Framework needs definitionRead
AI safety & alignment · Verification & proof search · Statistical inferenceOpen question

Can Finite Audits Certify an Adaptive Conversation?

Multi-turn certificates rely on worst-case transition and persistence properties, while practical audits observe only a finite selection of conversations. Efficiency payoff: Spend audits on safety-relevant states and transitions instead of repeatedly testing whole conversations without a coverage argument.

Editorially scoped

3 sources · Framework needs definitionRead
AI safety & alignment · Verification & proof searchOpen question

Can Hidden Backdoors Be Ruled Out?

A model can pass extensive evaluations while retaining a rare hidden failure mode. Understanding what evidence can rule one out is a foundational problem in AI assurance.

Editorially scoped

2 sources · Framework needs definitionRead
AI safety & alignment · Verification & proof searchOpen question

Certifying Jailbreak Resistance Without Blanket Refusal

A defense must protect against an entire specified perturbation set while remaining useful. Stronger detection can otherwise buy safety by rejecting most requests. Efficiency payoff: Reduce the number of extra generations or exhaustive edit checks needed for a meaningful robustness/utility certificate.

Editorially scoped

4 sources · Framework needs definitionRead
AI safety & alignment · Verification & proof searchOpen question

Controlling an Untrusted AI

A deployment protocol should remain reliable even when helpful behavior during evaluation does not predict behavior later.

Editorially scoped

3 sources · Framework needs definitionRead
AI safety & alignment · Statistical learning theory · Transfer & distribution shiftOpen question

Knowing When Not to Answer

High average accuracy is not enough when mistakes concentrate on the people or situations least visible to the evaluator.

Editorially scoped

5 sources · Framework needs definitionRead
AI safety & alignment · Embeddings & representationsOpen question

Learning Beyond the Teacher

Can a learner reliably become better than the source of its supervision? The answer matters whenever strong models are trained using weaker judges.

Editorially scoped

5 sources · Framework needs definitionRead
Transfer & distribution shift · Verification & proof search · AI safety & alignmentOpen question

Learning in Synthetic Worlds

A synthetic environment can execute perfectly while rewarding behavior that exploits its mismatch with the intended world.

Editorially scoped

3 sources · Framework needs definitionRead
AI safety & alignment · RL & post-training · RL foundations · Control & roboticsOpen question

Learning Without Catastrophic Experiments

An autonomous system may not be allowed to discover a dangerous action by trying it. Learning must use structure to distinguish safe expansion from reckless exploration.

Editorially scoped

3 sources · Framework needs definitionRead