Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
This paper addresses gradient conflicts in jointly training vision-language models (VLMs) with reinforcement learning by proposing Skill-Decoupled On-Policy Expert Distillation (SCOPED). SCOPED trains capability-specific experts independently and then distills explicit skills via reward-guided reflection into a unified student model, preventing the student from directly accessing these skills during inference. Empirically, SCOPED achieves state-of-the-art performance across eight multimodal benchmarks by effectively integrating independently acquired capabilities without reward-hacking biases.
Distilling explicit, reward-guided skills into VLMs without the student model ever seeing the skills themselves unlocks superior multimodal performance.
Reinforcement learning (RL) can elicit specialized visual capabilities in vision-language models (VLMs), yet jointly optimizing heterogeneous objectives often introduces cross-capability gradient conflicts. Training experts independently mitigates this interference, but direct reward optimization may entangle specialized knowledge with reward-hacking biases. This paper proposes Skill-Decoupled On-Policy Expert Distillation (SCOPED), which trains capability-specific experts and extracts explicit skills through reward-guided reflection. During on-policy distillation, these skills guide expert supervision while remaining unavailable to the unified student, encouraging it to internalize procedural knowledge without explicit skills at inference time. Across eight multimodal benchmarks, SCOPED consistently outperforms the backbone and mixed-capability RL, demonstrating the effectiveness of skill-decoupled on-policy expert distillation for integrating independently acquired capabilities. Further analyses reveal interference during joint capability optimization and suggest that skill augmentation promotes more consistent expert trajectories, greater attention to task-relevant visual evidence, and better preservation of specialized expertise in the unified student. All code and data are available at https://github.com/NEUIR/SCOPED.