Search papers, labs, and topics across Lattice.
This paper analyzes Direct Preference Optimization (DPO) as a statistical estimator of reward functions induced by a parametric policy class, demonstrating that DPO is misspecified when the true reward function is unrealizable within that class. The authors show that this misspecification leads to failure modes like preference reversal and reward worsening. To address this, they propose AuxDPO, a modification to the DPO loss function that introduces auxiliary variables to better approximate the two-stage RLHF solution.
DPO, a popular RLHF alternative, can actually *hurt* performance due to statistical misspecification, but a simple fix (AuxDPO) can bring it back on track.
Direct alignment algorithms such as Direct Preference Optimization (DPO) fine-tune models based on preference data, using only supervised learning instead of two-stage reinforcement learning with human feedback (RLHF). We show that DPO encodes a statistical estimation problem over reward functions induced by a parametric policy class. When the true reward function that generates preferences cannot be realized via the policy class, DPO becomes misspecified, resulting in failure modes such as preference order reversal, worsening of policy reward, and high sensitivity to the input preference data distribution. On the other hand, we study the local behavior of two-stage RLHF for a parametric class and relate it to a natural gradient step in policy space. Our fine-grained geometric characterization allows us to propose AuxDPO, which introduces additional auxiliary variables in the DPO loss function to help move towards the RLHF solution in a principled manner and mitigate the misspecification in DPO. We empirically demonstrate the superior performance of AuxDPO on didactic bandit settings as well as LLM alignment tasks.
