Search papers, labs, and topics across Lattice.
The paper introduces Multi-domain Contrastive Policy Optimization (MCPO) to improve the reasoning capabilities of Large Reasoning Models (LRMs) in multi-domain settings. MCPO uses contrastive learning to promote cross-domain knowledge sharing by identifying transferable reasoning trajectories as positive examples and incorrect rollouts as negative examples, while also consolidating in-domain knowledge. Experiments demonstrate that MCPO enhances LRM reasoning across multiple domains, even surpassing single-domain training in certain instances.
Cross-domain RL can actually *boost* reasoning in large models, if you use contrastive learning to transform harmful interference into beneficial knowledge transfer.
Post-training has significantly enhanced the reasoning capability of Large Reasoning Models (LRMs), especially with Reinforcement Learning (RL) like Group Relative Policy Optimization (GRPO). However, GRPO-style RL methods in multi-domain settings often fail to achieve consistent improvements across all domains due to inherent interference in policy optimization. Prior studies on multi-domain RL primarily focus on alleviating cross-domain interference, while often neglecting the pivotal role of knowledge sharing, which we argue is the key to transforming cross-domain interactions from harmful competition into beneficial transfer. To address this limitation, we propose Multi-domain Contrastive Policy Optimization (MCPO), which analyzes the structural relationships among rollouts and promotes cross-domain knowledge sharing and in-domain knowledge consolidation in a contrastive manner. Specifically, for a given prompt, MCPO identifies transferable reasoning trajectories from other domains as positive examples, while treating incorrect rollouts as negative ones. It then encourages consistent representations for positive pairs and pushes negative pairs apart, thereby facilitating knowledge transfer and reducing interference. Moreover, MCPO aligns intra-domain correct rollouts to build a consolidated representation space. In this way, MCPO contrastively learns a harmonious representation space that can accommodate diverse multi-domain knowledge. Empirical results show that MCPO improves the reasoning capabilities of LRMs across multiple domains and even outperforms single-domain training in some cases. Code is available at https://github.com/Maricalce/MCPO.
