Search papers, labs, and topics across Lattice.
This paper investigates the entangled roles of the coefficient $\beta$ in Direct Preference Optimization (DPO) for aligning language models, revealing that it simultaneously influences both the inverse preference-noise scale and the optimization dynamics. The authors demonstrate that the policy deviation achieved is non-monotonic with respect to $\beta$, leading to a "dead zone" at low values and a peak at intermediate values, complicating the interpretation of loss values across different $\beta$. To address these issues, they propose a centered-softplus reformulation of the DPO objective that disentangles these effects, allowing for independent tuning of the inverse preference-noise scale and learning rate.
The coefficient $\beta$ in DPO not only controls preference noise but also entangles optimization dynamics, leading to non-intuitive learning behaviors that can mislead model training.
Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient $\beta$ commonly interpreted as controlling the KL constraint to a reference policy. We show that $\beta$ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size. As a consequence, at a fixed learning rate the achieved policy deviation is non-monotone in $\beta$: it vanishes in a dead zone at small $\beta$, reaches a peak at an intermediate value, and decreases again for larger $\beta$. Moreover, standard DPO loss values are not comparable across $\beta$: runs with nearly identical loss curves can differ several-fold in KL divergence from the reference model. This entanglement obscures the role of $\beta$, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling. We propose a centered-softplus reformulation that is argmin-equivalent to DPO for $\beta>0$, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable. The normalized centered-softplus objective also admits a continuous $\beta\to0$ endpoint that reduces to a linear preference-margin objective.
