Search papers, labs, and topics across Lattice.
The paper introduces Confidence-Reward driven Preference Optimization (CRPO), a novel preference optimization method for machine translation that addresses the challenge of low-quality preference data in Direct Preference Optimization (DPO). CRPO enhances data selection by incorporating model confidence alongside reward scores, focusing on challenging sentence pairs where the model exhibits uncertainty or poor performance. Experiments demonstrate that CRPO outperforms existing preference optimization techniques, including RS-DPO, RSO, and MBR score, in both translation accuracy and data efficiency across LLMs and encoder-decoder models like NLLB.
By focusing on the most challenging examples, CRPO significantly boosts machine translation accuracy and data efficiency compared to standard preference optimization techniques.
Large language models (LLMs) have shown great potential in natural language processing tasks, but their application to machine translation (MT) remains challenging due to pretraining on English-centric data and the complexity of reinforcement learning from human feedback (RLHF). Direct Preference Optimization (DPO) has emerged as a simpler and more efficient alternative, but its performance depends heavily on the quality of preference data. To address this, we propose Confidence-Reward driven Preference Optimization (CRPO), a novel method that combines reward scores with model confidence to improve data selection for fine-tuning. CRPO selects challenging sentence pairs where the model is uncertain or underperforms, leading to more effective learning. While primarily designed for LLMs, CRPO also generalizes to encoder-decoder models like NLLB, demonstrating its versatility. Empirical results show that CRPO outperforms existing methods such as RS-DPO, RSO and MBR score in both translation accuracy and data efficiency.
Amazon Science
Tsinghua AI