Can Behavior Reveal the Reward?
Matching an expert's actions need not reveal why they act that way. Reliable reward learning must describe what behavior can and cannot tell us.
Editorially scoped
THEORIA / RESEARCH CATALOG
Find your next research direction. Explore conjectures and open questions through their models, mathematical targets and connections.
Explore with /next-steps ↗ or /attack slash commands
A source-linked collection of theoretical open problems. Explore challenges in AI safety, learning theory, transformers, world models, optimization, memory and inference. Each entry gives a precise research target, assumptions and known progress. Open status is provisional; broader programs explicitly need a framework. Sources & methodology ↗
Seed catalog · Browse the bundled collection. Search matches words; Supabase mode uses ranked full-text search.
No selection includes all.
No selection includes all.
No selection includes all.
Choose multiple options. Results match any selected value within each filter.
12 problems
Matching an expert's actions need not reveal why they act that way. Reliable reward learning must describe what behavior can and cannot tell us.
Editorially scoped
Multi-turn certificates rely on worst-case transition and persistence properties, while practical audits observe only a finite selection of conversations. Efficiency payoff: Spend audits on safety-relevant states and transitions instead of repeatedly testing whole conversations without a coverage argument.
Editorially scoped
Neural networks can represent complex interactions parsimoniously. Turning their learned features into a trustworthy scientific interaction graph requires more than a good fit.
Editorially scoped
The information that determines a model need not be information an algorithm can use efficiently. Higher-order observations may trade measurement cost for computational tractability.
Editorially scoped
Observed correlations reflect both interactions and the dynamics used to observe them. Inference near slow or critical regimes must account for both.
Editorially scoped
Search engines learn from behavior they helped create. Without identifying the exposure mechanism, more clicks can reinforce an incorrect ranking.
Editorially scoped
Near separability, tiny changes in regularization or sample size can have large effects on coefficients and confidence even in this simple convex model. Efficiency payoff: Guide finite-data regularization and stopping choices near a sharp statistical transition instead of relying on unstable asymptotic extrapolations.
Editorially scoped
A learned interaction graph can confuse direct dependence with unobserved causes. Hidden-spin models make this ambiguity mathematically concrete.
Editorially scoped
More data is not always better value. A learner should discover which sources provide the most useful evidence for the target task.
Editorially scoped
Two distributions with equally large support can require very different amounts of data to distinguish from a reference. A sharp instance-dependent bound would capture that difference.
Editorially scoped
A precise posterior can still be confidently wrong. Useful Bayesian learning needs uncertainty that remains informative when both the model and inference are approximate.
Editorially scoped
A compact equation fitted to coarse observations may describe only the discretization or unresolved fluctuations, rather than the underlying constitutive law. Efficiency payoff: Identify when a small reusable PDE model can replace expensive field-level prediction without requiring unnecessarily fine observations.
Editorially scoped