Search papers, labs, and topics across Lattice.
This paper investigates three fusion paradigms鈥擬erge, Mix RL, and multi-teacher on-policy distillation (MOPD)鈥攆or consolidating reinforcement learning with verifiable rewards (RLVR) capabilities across multiple domains. By comparing these methods using shared experts and a multi-domain benchmark suite, the authors reveal that while average performance differences are minimal, significant variations exist at the domain level, particularly highlighting the importance of task-vector geometry. The findings provide actionable guidelines for selecting the appropriate fusion method based on specific training conditions and goals, such as prioritizing expert utilization or domain-specific gains.
Merging existing expert capabilities can yield significant performance boosts, but choosing the right fusion method can make all the difference in multi-domain reinforcement learning.
Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate domain experts and subsequently consolidating them. We organize three fusion paradigms by the artefacts they reuse: Merge combines expert task vectors, Mix RL pools their datasets, and multi-teacher on-policy distillation (MOPD) uses both. Because they have largely been studied in isolation, how they compare and how to choose among them remain unclear. We compare all three using shared experts and data across model scales and a multi-domain benchmark suite. Although their average performance differs by at most 1.4 points, the gap reaches 8.6 points on a single benchmark, with domain-level variation tracking cross-domain relations visible in task-vector geometry. Training dynamics expose distinct constraints: Mix RL depends on domain mixture proportions, MOPD remains bounded by its teachers, and Merge compresses all expert updates into one. All three improve single-sample accuracy without measurable gains in solution coverage or losses in held-out capabilities. These results yield a practical guideline: use Merge when experts already exist and cheap fusion is paramount; Mix RL when training a unified model without experts, with domain proportions adjusted for cross-domain transfer; and MOPD when preserving domain-specific gains matters more than surpassing teachers or minimizing end-to-end cost.
