Search papers, labs, and topics across Lattice.
The paper identifies a theoretical inconsistency in standard preference learning methods like DPO, showing they yield vacuous generalization guarantees for equicontinuous hypothesis sets common in LLMs. To address this, they propose a margin-shifted ranking framework and derive structure-aware H-consistency bounds, adapting the margin based on the semantic distance between responses. Their novel objective, SA-DPO, improves consistency, and they demonstrate that heavy-tailed surrogate losses offer superior consistency guarantees for capacity-bounded models compared to the standard logistic loss.
Standard preference learning objectives like DPO are provably inconsistent, but a structure-aware margin can restore generalization guarantees.
Preference learning has become the foundation of aligning Large Language Models (LLMs) with human intent. Popular methods, such as Direct Preference Optimization (DPO), minimize surrogate losses as proxies for the intractable pairwise ranking loss. However, we demonstrate that for the equicontinuous hypothesis sets typical of neural networks, these standard surrogates are theoretically inconsistent, yielding vacuous generalization guarantees. To resolve this, we formulate LLM alignment within a margin-shifted ranking framework. We derive rigorous $H$-consistency bounds that depend on enforcing a separation margin $纬$. Crucially, we extend this to Structure-Aware $H$-consistency, introducing a novel objective (SA-DPO) that adapts the margin based on the semantic distance between responses to handle synonyms and hard pairs. Finally, we analyze the trade-off between consistency and model limitations via the Margin-Capacity Profile, proving that heavy-tailed surrogates (such as the Polynomial Hinge family) offer superior consistency guarantees for capacity-bounded models compared to the standard logistic loss used in DPO.
