Search papers, labs, and topics across Lattice.
This paper presents an incentive-score decomposition of preference optimization objectives used in LLM alignment, showing that they share identical local update directions but differ in weighting coefficients. They identify a "disentanglement band" (DB) condition that characterizes when training avoids likelihood displacement by suppressing the rejected response while maintaining the chosen one. Based on the DB, they propose a reward calibration (RC) method that adaptively rebalances chosen/rejected updates, demonstrating improved disentanglement and downstream performance across objectives.
Preference optimization objectives, despite their diversity, can be steered towards disentangled dynamics that avoid suppressing the chosen response alongside the rejected one, simply by satisfying a "disentanglement band" condition.
Preference optimization is widely used to align large language models (LLMs) with human preferences. However, many margin-based objectives suppress the chosen response along with the rejected one, a phenomenon known as likelihood displacement, and no general mechanism currently prevents this across objectives. We bridge this gap by presenting a unified \emph{incentive-score decomposition} of preference optimization, revealing that diverse objectives share identical local update directions and differ only in their scalar weighting coefficients. Building on this decomposition, by analyzing the dynamics of the chosen/rejected likelihoods, we identify the \emph{disentanglement band} (DB), a simple, testable condition that characterizes when training can avoid likelihood displacement by realizing the preferred pathway: suppressing the loser while maintaining the winner, possibly after an initial transient. Leveraging the DB, we propose a plug-and-play \emph{reward calibration} (RC) that adaptively rebalances chosen versus rejected updates to satisfy the DB and mitigate likelihood displacement, without redesigning the base objective. Empirical results show that RC steers training toward more disentangled dynamics and often improves downstream performance across a range of objectives. Our code is available at https://github.com/IceyWuu/DisentangledPreferenceOptimization.
