Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of reward hacking in reinforcement learning from human feedback (RLHF) by introducing a distributional reward model that quantifies uncertainty in proxy reward models. By applying Bayesian inference and KL-distributionally robust optimization, the authors derive a closed-form effective reward that integrates various existing heuristics for reward aggregation, such as mean aggregation and worst-case optimization. The key result reveals that these heuristics can be unified under a single framework, enhancing our understanding of their underlying assumptions and improving the robustness of RLHF systems.
Reward hacking can be mitigated by a unified distributional reward model that reveals the hidden connections between existing aggregation heuristics in RLHF.
Reinforcement learning from human feedback (RLHF) is bottlenecked by \emph{reward hacking}, where the policy exploits errors in a proxy reward model (RM) and produces high RM scores without genuine quality gains. A natural mitigation is \emph{pessimism}: penalizing rewards in regions where the RM is uncertain. However, standard scalar RMs provide no principled notion of uncertainty. We argue that the right object is a \emph{distributional} reward model $p(r\mid x,y)$. Under either a Bayesian inference or a KL-distributionally robust optimization (KL-DRO) lens, the KL-regularized RLHF objective admits a closed-form effective reward $\tilde r(x,y) = \pmβ\log\mathbb{E}_p[e^{\pm r/β}]$. The pessimistic branch unifies the prior heuristics for RM ensemble aggregation: mean aggregation, worst-case optimization (WCO), and uncertainty-weighted optimization (UWO) all emerge as limits or truncations of this single expression. This also clarifies the implicit assumptions of each existing rule.