Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper identifies a theoretical misalignment in the Direct Preference Optimization (DPO) loss function, arguing that its indefinite maximization of logits differences leads to training instability and reward hacking. To address this, the authors propose a novel loss function derived directly from the RLHF optimality condition that targets a specific, finite value for the logits difference. They demonstrate theoretically that their method avoids large gradients associated with DPO, and empirically validate its effectiveness by fine-tuning a Qwen2.5-7B model, achieving improved win rates over DPO and competitive performance against Llama-3.1-8B.
DPO's instability problem is solved with a new loss function that directly targets a finite logits difference, leading to better alignment and preventing reward hacking.
The alignment of large language models (LLMs) with human preferences is commonly achieved through Reinforcement Learning from Human Feedback (RLHF). Direct Preference Optimization (DPO) simplified this paradigm by establishing a direct mapping between the optimal policy and a reward function, eliminating the need for an explicit reward model. However, we argue that the DPO loss function is theoretically misaligned with its own derivation, as it promotes the indefinite maximization of a logits difference, which can lead to training instability and reward hacking. In this paper, we propose a novel loss function derived directly from the RLHF optimality condition. Our proposed loss targets a specific, finite value for the logits difference, which is dictated by the underlying reward, rather than its maximization. We provide a theoretical analysis, including a gradient-based comparison, to demonstrate that our method avoids the large gradients that plague DPO when the probability of dispreferred responses approaches zero. This inherent stability prevents reward hacking and leads to more effective alignment. We validate our approach by fine-tuning a Qwen2.5-7B model, showing significant win-rate improvements over a standard DPO baseline and achieving competitive performance against larger models like Llama-3.1-8B.