Search papers, labs, and topics across Lattice.
This paper introduces Contrastive Policy Optimization (CPO), a novel method for advantage shaping in reinforcement learning with verifiable rewards (RLVR) that addresses the limitations of entropy as a correctness signal. By utilizing token-level contrastive disagreement between reference-guided and vanilla generation distributions, CPO effectively differentiates between useful uncertainty and detrimental confusion, leading to improved performance on various benchmarks. Empirical results demonstrate that CPO not only outperforms traditional entropy-based methods but also resolves the zero-advantage problem, enhancing both exploration and exploitation strategies in reinforcement learning.
Contrastive Policy Optimization reveals that distinguishing between correct and incorrect responses can significantly enhance reinforcement learning performance.
Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correctness signal. We propose Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping. Both theoretical and empirical results show that this disagreement reliably indicates token-level correctness. We further show that On-policy Distillation is a special case of CPO, where the posterior distribution is instantiated by an external teacher model. CPO also resolves the zero-advantage problem. Experiments on in-domain and out-of-domain benchmarks demonstrate that CPO substantially outperforms entropy-based RLVR methods while maintaining strong generalization. Further analysis shows that correct and incorrect responses naturally support exploration and exploitation respectively, and balancing both leads to the best performance.
