Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper provides a theoretical unification of preference learning methods for aligning LLMs, demonstrating that methods like RLHF, DPO, IPO, KTO, and SimPO can be understood through three orthogonal axes: preference model, regularization mechanism, and data distribution. It formalizes these axes with definitions and theorems, revealing the coverage separation between online and offline methods, scaling laws for reward overoptimization, and failure conditions for direct alignment. The analysis identifies how specific design choices lead to failure modes like length hacking and mode collapse, and it synthesizes empirical findings into a practitioner's decision guide.
Preference learning methods like RLHF and DPO are not as different as you think: they're just different choices along three key axes.
Aligning large language models (LLMs) with human preferences has become essential for safe and beneficial AI deployment. While Reinforcement Learning from Human Feedback (RLHF) established the dominant paradigm, a proliferation of alternatives -- Direct Preference Optimization (DPO), Identity Preference Optimization (IPO), Kahneman-Tversky Optimization (KTO), Simple Preference Optimization (SimPO), and many others -- has left practitioners without clear guidance on method selection. This survey provides a \textit{theoretical unification} of preference learning methods, revealing that the apparent diversity reduces to principled choices along three orthogonal axes: \textbf{(I) Preference Model} (what likelihood model underlies the objective), \textbf{(II) Regularization Mechanism} (how deviation from reference policies is controlled), and \textbf{(III) Data Distribution} (online vs.\ offline learning and coverage requirements). We formalize each axis with precise definitions and theorems, establishing key results including the coverage separation between online and offline methods, scaling laws for reward overoptimization, and conditions under which direct alignment methods fail. Our analysis reveals that failure modes -- length hacking, mode collapse, likelihood displacement -- arise from specific, predictable combinations of design choices. We synthesize empirical findings across 50+ papers and provide a practitioner's decision guide for method selection. The framework transforms preference learning from an empirical art into a theoretically grounded discipline.