Search papers, labs, and topics across Lattice.
This paper introduces Se-DPO, a novel mechanism for Direct Preference Optimization (DPO) that dynamically adjusts token credit based on each token's contribution to preference outcomes during training. By recognizing that the importance of tokens evolves, Se-DPO calibrates token credit using the model's internal signals, enhancing the reliability of the reward signal. Experimental results demonstrate that Se-DPO outperforms traditional DPO methods, achieving significant improvements in evaluation metrics on benchmark tasks.
Static token credit misaligns with evolving training dynamics, but Se-DPO's adaptive approach boosts performance by nearly 10 points on key benchmarks.
Direct Preference Optimization (DPO) aggregates token-level log-probability ratios via uniform summation, implicitly treating all tokens as contributing equally to the preference signal. However, the contribution of individual tokens to the preference signal varies. We introduce token credit, which modulates each token's KL regularization based on its contribution to the preference outcome. We derive that effective token credit is proportional to the magnitude of each token's implicit reward, and observe that this quantity evolves substantially during training. This implies that static token credit becomes increasingly misaligned as training progresses. In this work, we propose Se-DPO (Self-Evolving Token Credit for DPO), a live mechanism that derives token credit from the model's own evolving internal signals during DPO training. Since the reward signal varies in reliability across positions, Se-DPO calibrates token credit based on both the strength and the confidence of each token's contribution. Se-DPO requires no external models, adding only a lightweight calibration network with minimal computational overhead. Experiments show that Se-DPO improves over DPO by up to 9.8 points on AlpacaEval~2 and 12.2 points on Arena-Hard.
