AI safety & alignment · RL & post-training / Open question
Alignment Without Reward Hacking
Optimizing a learned reward can exploit its blind spots. Alignment needs guarantees that survive the distribution shift created by optimization itself.
Seed catalog · Showing the bundled record.
Problem statement
In a finite contextual bandit, let utility generate comparisons with up to additive error , where . Policies are softmax models and the fitted reward is a bounded linear feature model. Under paid adaptive comparisons, bound for while policy and reward are jointly updated, including coverage, misspecification and regularization costs.
Assumptions and scope
- Fix a stationary latent utility, a probabilistic comparison model and bounded contamination or approximation error; human preferences need not be perfectly represented by a scalar reward outside this model.
- Specify policy and reward classes, comparison access, coverage and the exact update rule. Held-out error on the initial distribution does not certify optimized policies.
- Compare with the best policy attainable under the same feedback budget, and state whether utility is bounded or has controlled tails.
What would settle it?
A finite-budget utility-regret bound and adaptive acquisition/regularization rule with a matching misspecification or coverage lower bound for the joint policy–reward process.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Chi-squared preference optimization, RPO and POWER already give offline robustness under coverage assumptions; catastrophic Goodhart examples expose KL limitations. The residual target is cost-optimal adaptive preference collection with jointly changing models and misspecification, not the existence of an anti-overoptimization algorithm. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Huang et al. — Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization ↗partial theory · source checked 2026-11-19
Abstract checked: KL regularization is too weak offline; χ²-regularized χPO attains single-policy-concentrability guarantees.
- Liu et al. — Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer ↗partial theory · source checked 2026-11-19
Abstract checked: adversarial-reward pessimism with partial-coverage sample efficiency, implemented as DPO plus SFT loss.
- Rashidinejad and Tian — Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking ↗partial theory · source checked 2026-11-19
Abstract checked: two statistical reward-hacking types in offline preference optimization; finite-sample guarantees competing with the best covered policy.
- Kwa, Thomas and Garriga-Alonso — Catastrophic Goodhart ↗partial theory · source checked 2026-11-19
Abstract checked: with heavy-tailed reward error, KL-regularized optima can attain arbitrarily high proxy reward with no utility gain; light tails are benign.