Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces SLIME (Stabilized Likelihood Implicit Margin Enforcement), a novel reference-free alignment objective for preference optimization in LLMs that addresses the objective mismatch in existing methods like DPO. SLIME decouples preference learning from generation quality by incorporating an anchoring term to maximize the likelihood of preferred responses, a stabilizing penalty to prevent rejected token probabilities from collapsing, and a dual-margin mechanism for boundary shaping. Experiments demonstrate that SLIME outperforms state-of-the-art baselines while maintaining higher generation stability, mitigating issues like unlearning and formatting collapse.
Stop LLMs from unlearning: SLIME stabilizes likelihood during preference optimization, preventing degradation of high-quality outputs and formatting collapse.
Direct preference optimization methods have emerged as a computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF) for aligning Large Language Models (LLMs). Latest approaches have streamlined the alignment process by deriving implicit reward functions, yet they often suffer from a critical objective mismatch: optimizing the relative margin between chosen and rejected responses does not guarantee the preservation of the chosen response's absolute likelihood. This can lead to unlearning, where the model degrades the probability of high-quality outputs to satisfy margin constraints, and formatting collapse caused by the over-penalization of rejected sequences. In this work, we introduce SLIME (Stabilized Likelihood Implicit Margin Enforcement), a reference-free alignment objective designed to decouple preference learning from generation quality. SLIME incorporates a three-pronged objective: (1) an anchoring term to maximize the likelihood of preferred responses; (2) a stabilizing penalty that prevents the probabilities of rejected tokens from collapsing to zero; and (3) a dual-margin mechanism that combines hard and soft constraints for precise boundary shaping. Our results demonstrate that SLIME achieves superior performance compared to state-of-the-art baselines while maintaining higher generation stability.