Search papers, labs, and topics across Lattice.
This paper introduces Contrastive On-Policy Distillation (COPD), a novel framework that enhances student model reasoning by employing a contrastive approach to token-level supervision. By leveraging a frozen teacher model to score student outputs under varying reasoning instructions, COPD provides a direct signal for optimizing reasoning strategies rather than merely matching token distributions. Experimental results across nine multimodal benchmarks demonstrate that COPD significantly reduces reasoning length while improving efficiency and maintaining model performance across various tasks and scales.
Token-level contrastive signals can streamline reasoning processes, leading to more efficient AI models without sacrificing performance.
On-policy Distillation (OPD) supervises a student model on trajectories sampled from its own policy by minimizing the divergence between the output distributions of the teacher and student at each token position, thereby providing dense token-level supervision. Although existing OPD methods have demonstrated strong performance in improving the reasoning ability of student models, their objectives fundamentally rely on token-level distribution matching. Consequently, they lack an explicit signal for comparing a token's relative compatibility across reasoning modes and thus do not directly model preferences between these modes. To address this limitation, we propose COPD, a contrastive OPD framework. Specifically, for each token generated by the student model, a frozen teacher model scores the same student state under two contrasting instructions that elicit light and heavy reasoning. The difference between the resulting log probabilities serves as a token-level advantage signal to guide the OPD update. Rather than merely imitating a single teacher distribution, COPD directly encourages the student model to learn more concise and efficient reasoning strategies. We conduct experiments on nine multimodal benchmarks covering both reasoning and understanding tasks. The results show that COPD substantially reduces reasoning length without compromising model performance and consistently improves efficiency across different tasks and model scales. Furthermore, the contrastive formulation can be seamlessly integrated into the On-policy Self-distillation (OPSD) framework, where self-contrastive supervision is constructed without an additional teacher model, thereby enabling the model to distill itself toward lightweight reasoning.
