Search papers, labs, and topics across Lattice.
This paper introduces Contrastive Reinforced Policy Optimization (CRPO), a novel approach that reformulates On-Policy Self-Distillation (OPSD) through a contrastive learning lens to address the exposure bias inherent in traditional OPSD methods. By employing predictive entropy to differentiate between positive and negative positions, CRPO enhances the optimization process, leading to improved training stability and generalization in multi-turn reasoning tasks. Extensive evaluations across 13 benchmarks reveal that CRPO outperforms existing reinforcement learning and self-distillation techniques, marking a significant advancement in agentic learning frameworks.
CRPO effectively mitigates exposure bias in self-distillation, leading to superior performance in complex reasoning tasks.
Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD). While OPSD provides dense, logit-level supervision, it inherently suffers from exposure bias due to the privileged information of the self-teacher. In multi-turn agentic settings, this leads to reasoning route convergence and the loss of clear optimization directions. To tackle these challenges, we introduce Contrastive Reinforced Policy Optimization (CRPO), which reformulates agentic OPSD from a contrastive learning perspective. By leveraging predictive entropy to distinguish between positive positions (reflective exploration) and negative positions (exposure bias), CRPO conducts group-wise contrast to preserve reliable, fine-grained optimization signals. Extensive evaluations across 13 challenging reasoning and deep-search benchmarks demonstrate that CRPO consistently outperforms existing reinforcement learning and self-distillation baselines, significantly enhancing training stability and generalization in long-horizon interactions.
