← All AI/ML open problems

AI safety & alignment · RL & post-training / Open question

Alignment Without Reward Hacking

Optimizing a learned reward can exploit its blind spots. Alignment needs guarantees that survive the distribution shift created by optimization itself.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

In a finite contextual bandit, let utility u(x,a)∈[0,1]u(x,a)\in[0,1] generate comparisons with Pr⁡(a≻b∣x)=σ(u(x,a)−u(x,b))\Pr(a\succ b\mid x)=\sigma(u(x,a)-u(x,b)) up to additive error η\eta, where σ(z)=(1+e−z)−1\sigma(z)=(1+e^{-z})^{-1}. Policies are softmax models and the fitted reward is a bounded linear feature model. Under BB paid adaptive comparisons, bound J(π⋆)−J(π^)J(\pi^\star)-J(\hat\pi) for J(π)=Ex,a∼πu(x,a)J(\pi)=\mathbb E_{x,a\sim\pi}u(x,a) while policy and reward are jointly updated, including coverage, misspecification and regularization costs.

Assumptions and scope

  • Fix a stationary latent utility, a probabilistic comparison model and bounded contamination or approximation error; human preferences need not be perfectly represented by a scalar reward outside this model.
  • Specify policy and reward classes, comparison access, coverage and the exact update rule. Held-out error on the initial distribution does not certify optimized policies.
  • Compare with the best policy attainable under the same feedback budget, and state whether utility is bounded or has controlled tails.

What would settle it?

A finite-budget utility-regret bound and adaptive acquisition/regularization rule with a matching misspecification or coverage lower bound for the joint policy–reward process.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Chi-squared preference optimization, RPO and POWER already give offline robustness under coverage assumptions; catastrophic Goodhart examples expose KL limitations. The residual target is cost-optimal adaptive preference collection with jointly changing models and misspecification, not the existence of an anti-overoptimization algorithm. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Huang et al. — Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization ↗partial theory · source checked 2026-11-19

    Abstract checked: KL regularization is too weak offline; χ²-regularized χPO attains single-policy-concentrability guarantees.

  2. Liu et al. — Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer ↗partial theory · source checked 2026-11-19

    Abstract checked: adversarial-reward pessimism with partial-coverage sample efficiency, implemented as DPO plus SFT loss.

  3. Rashidinejad and Tian — Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking ↗partial theory · source checked 2026-11-19

    Abstract checked: two statistical reward-hacking types in offline preference optimization; finite-sample guarantees competing with the best covered policy.

  4. Kwa, Thomas and Garriga-Alonso — Catastrophic Goodhart ↗partial theory · source checked 2026-11-19

    Abstract checked: with heavy-tailed reward error, KL-regularized optima can attain arbitrarily high proxy reward with no utility gain; light tails are benign.