Search papers, labs, and topics across Lattice.
This paper introduces Language-Routed Policy Optimization (LRPO), a novel online policy optimization framework that treats language selection as a learnable variable within multilingual reinforcement learning. LRPO elicits multilingual rollouts for each training question and uses their relative quality to inform preference-based policy updates, thereby increasing the diversity and informativeness of training signals. Experiments demonstrate that LRPO consistently improves multilingual performance by adaptively routing languages via a multi-armed bandit, effectively exploiting cross-lingual knowledge.
Unleashing multilingual rollouts and adaptively routing languages during policy optimization can significantly boost performance, proving that language diversity is a valuable, but often overlooked, training signal.
Large language models~(LLMs) are trained on heterogeneous multilingual corpora, yet existing policy optimization methods often implicitly restrict each training question to a single response language or rely on a fixed dominant language for supervision. We propose language-routed policy optimization (LRPO), an online policy optimization framework that treats language as a selectable variable. LRPO elicits multilingual rollouts for each training question and integrates their relative quality into preference-based policy updates, increasing the diversity and informativeness of training signals under the fixed rollout budget. To adaptively determine which languages to explore during reinforcement learning, we introduce a trainable language router formulated as a multi-armed bandit, balancing exploration of underutilized languages with exploitation of more informative ones. Extensive experiments show that LRPO consistently improves multilingual performance, demonstrating that adaptive language routing enables effective cross-lingual knowledge exploitation for training. We release all the resources at https://github.com/Guochry/LRPO.
