Search papers, labs, and topics across Lattice.
This paper introduces CroCo, a method for cross-lingual contrastive preference tuning of LLMs using self-generated responses. It leverages a reward model trained on English preferences to rank responses in multiple languages, enabling effective preference tuning without language-specific annotations. Experiments across 14 languages demonstrate that CroCo improves performance on both structured tasks and open-ended generation, while also mitigating catastrophic forgetting.
Train your reward model on English preferences and unlock effective preference tuning across a diverse range of languages, even without language-specific annotations.
Prior work establishes that controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in English. We extend this method to multiple languages and evaluate two models across a total of 14 high and low-resource languages on a diverse set of tasks. Our central finding is that cross-lingual contrastive preference tuning on self-generations (CroCo) transfers without language-specific preference annotation. A reward model trained on English preferences (atop a multilingual base) produces useful within-language rankings across most languages, and pairing in either a monolingual or multilingual setting improves over each model on the majority of setups while preventing the catastrophic forgetting of supervised fine-tuning. We observe that the gains require on-policy data. Off-policy responses reduce the benefit and online preference optimization fails to improve over the offline variant. Specifically, on structured tasks, our method matches or exceeds the base in 6/7 languages for EuroLLM-9B and 4/7 settings for Aya-3B. On open-ended generation, both tuned models win against their respective base across 11 evaluated languages. Overall, we show promising directions for multilingual preference tuning.
