Search papers, labs, and topics across Lattice.
This paper introduces Mult-DPO, a novel optimization objective that extends direct preference optimization (DPO) from pairwise to set-wise preferences in recommender systems, addressing the challenge of user feedback that typically involves multiple positive items without a prescribed order. By leveraging a multinomial surrogate likelihood, Mult-DPO enables efficient alignment of large language models (LLMs) with user preferences, while maintaining a tractable closed-form objective. The authors demonstrate that this approach provides a tractable upper bound on the marginalized Plackett-Luce DPO loss, offering insights into optimizing against set-wise preference data with varying levels of difficulty among negative samples.
Mult-DPO reveals that a multinomial approach can effectively align LLMs with complex user preferences, outperforming traditional pairwise methods.
Direct preference optimization (DPO) is a simple and effective alignment strategy for large language models (LLMs) based on pairwise preferences. In recommender systems, however, user feedback is rarely pairwise. For a given context, e.g., a user, a session, or a conversation, we typically observe set-wise preferences with multiple positive items, where every positive item should outrank every unobserved or explicitly negative item, with no prescribed order among the positives or the negatives themselves. A natural generalization is to use the Plackett-Luce (PL) reward model, which extends the Bradley-Terry reward model underlying vanilla DPO from pairwise preferences to full rankings of candidates. However, we show that adapting the PL model to set-wise preferences requires marginalizing over all positive orderings, where the resulting expression is combinatorial in complexity. To address this fundamental challenge, we propose Mult-DPO, a novel DPO objective with a tractable multinomial surrogate likelihood over set-wise preference events for the user-preference alignment of LLM-based recommender systems. The multinomial construction is not itself a ranking distribution, but it is defined on the same reward-induced weight space and admits a closed-form DPO-style objective, enabling direct alignment of LLMs with multiple candidates through a classification-style objective. In addition, we prove that the multinomial DPO loss is a tractable upper bound on the marginalized PL DPO loss when optimizing against the set-wise preference data. We further characterize the tightness of this bound in terms of the relative total weight of positives versus negatives, which provides insights into tightening the bound with richer or harder negatives. Finally, we extend Mult-DPO to the alignment of LLMs with multiple preference levels. Code is available at https://github.com/yaochenzhu/Mult_DPO
