← All AI/ML open problems

Inverse RL & imitation · Statistical inference · AI safety & alignment / Open question

Can Behavior Reveal the Reward?

Matching an expert's actions need not reveal why they act that way. Reliable reward learning must describe what behavior can and cannot tell us.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

In finite discounted MDPs, let rθ(s,a)=θ⊤ϕ(s,a)r_\theta(s,a)=\theta^\top\phi(s,a), ∥θ∥2≤1\|\theta\|_2\leq1, and let the demonstrator follow an entropy-regularized optimal policy with unknown τ∈[τmin⁡,τmax⁡]\tau\in[\tau_{\min},\tau_{\max}]. Transitions satisfy sup⁡s,a∥P(⋅∣s,a)−P0(⋅∣s,a)∥TV≤η\sup_{s,a}\|P(\cdot\mid s,a)-P_0(\cdot\mid s,a)\|_{\mathrm{TV}}\leq\eta for known P0P_0, and only emissions through a known, possibly noninvertible channel are logged. From nn trajectories determine confidence regions for rewards modulo observationally equivalent behavior, retaining shaping and reward-temperature scale ambiguities.

Assumptions and scope

  • Fix the environment and observation families and an explicit restricted demonstrator model; arbitrary behavior does not identify an arbitrary internal reward.
  • Quotient out the transformations preserving the observed behavior, including applicable reward shifts, shaping or temperature–scale ambiguities; normalization is not evidence of unique intent.
  • Distinguish imitation accuracy, reward-parameter error and correct decisions in new tasks, with expert-data and environment-information costs explicit.

What would settle it?

An identifiable quotient or partial-identification region with finite-trajectory uncertainty bounds under a stated imperfect-demonstrator model, together with observational-equivalence and misspecification lower bounds.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Skalse and Abate already characterize ambiguity and misspecification for standard behavioral models; Cao et al. give entropy-regularized identification results; fast IRL rates exist for linear rewards under stated assumptions. The residual target couples partial observation and decision-model uncertainty rather than rediscovering reward ambiguity. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Skalse and Abate — Partial Identifiability and Misspecification in Inverse Reinforcement Learning ↗partial theory · source checked 2026-11-19

    Primary abstract checked: ambiguity and misspecification robustness are characterized for common behavioral models. Basic reward ambiguity is an established result, not a new conjecture.

  2. Cao, Cohen and Szpruch — Identifiability in Inverse Reinforcement Learning ↗partial theory · source checked 2026-11-19

    Primary search abstract checked: entropy-regularized reward equivalence; sufficiently different environments or discount factors can identify rewards up to a constant under the stated conditions.

  3. Schlaginhaufen and Kamgarpour — Fast Rates for Inverse Reinforcement Learning ↗partial theory · source checked 2026-11-19

    Primary abstract checked: entropy-regularized Min-Max-IRL, O(1/n) trajectory-KL and Hessian-norm parameter rates for linear rewards, with a local minimax result in the well-specified deterministic case; misspecification and general Borel spaces are also addressed.