Search papers, labs, and topics across Lattice.
The paper introduces Contrastive Learning in Policy Optimization (CLIPO) to improve Reinforcement Learning with Verifiable Rewards (RLVR) for LLMs by addressing the issue of training on process-wrong but outcome-correct rollouts. CLIPO incorporates a contrastive loss over successful rollouts, encouraging the LLM to learn invariant structures across correct reasoning paths. Experiments show that CLIPO enhances generalization and robustness across various reasoning benchmarks compared to RLVR baselines, mitigating step-level reasoning inconsistencies and suppressing hallucinations.
LLMs can be steered away from hallucination and towards more robust reasoning by using contrastive learning to capture the shared structure of successful reasoning paths, even when the final answer is correct.
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capacity of Large Language Models (LLMs). However, RLVR solely relies on final answers as outcome rewards, neglecting the correctness of intermediate reasoning steps. Training on these process-wrong but outcome-correct rollouts can lead to hallucination and answer-copying, severely undermining the model's generalization and robustness. To address this, we incorporate a Contrastive Learning mechanism into the Policy Optimization (CLIPO) to generalize the RLVR process. By optimizing a contrastive loss over successful rollouts, CLIPO steers the LLM to capture the invariant structure shared across correct reasoning paths. This provides a more robust cross-trajectory regularization than the original single-path supervision in RLVR, effectively mitigating step-level reasoning inconsistencies and suppressing hallucinatory artifacts. In experiments, CLIPO consistently improves multiple RLVR baselines across diverse reasoning benchmarks, demonstrating uniform improvements in generalization and robustness for policy optimization of LLMs. Our code and training recipes are available at https://github.com/Qwen-Applications/CLIPO.
DAMO