Search papers, labs, and topics across Lattice.
This paper introduces Cross-Lingual Ranking Preference Optimization (CRPO), a framework that enhances the alignment of Large Language Models (LLMs) by utilizing English-centric preference data to improve performance in other languages. By implementing a hierarchical structure for parallel preference pairs, CRPO optimizes both intra- and inter-lingual preferences, leading to superior instruction-following and knowledge utilization across five languages. The results demonstrate that CRPO not only outperforms traditional methods but also stabilizes the preference manifold, significantly improving reward margins and the likelihood of desirable outputs.
CRPO reveals that leveraging English preference data can dramatically enhance multilingual LLM performance, outperforming standard methods across diverse languages.
The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-Lingual Ranking Preference Optimization (CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment.
