Search papers, labs, and topics across Lattice.
The paper introduces Dual Consensus Reinforcement Learning (DCRL), a self-supervised method for improving LLM reasoning by mitigating the issue of models converging on spurious majority answers in label-free RLVR. DCRL employs a two-stage consensus mechanism where the model acts as both an anchor, generating dominant responses, and an explorer, generating diverse auxiliary signals via temporary unlearning. By using the harmonic mean of these two signal sets as the training target, DCRL achieves more reliable learning signals and escapes the spurious majority trap, leading to improved performance and stable training.
LLMs can escape the trap of converging on popular but incorrect answers in unsupervised RLVR by temporarily "unlearning" and exploring diverse response options.
Current label-free RLVR approaches for large language models (LLMs), such as TTRL and Self-reward, have demonstrated effectiveness in improving the performance of LLMs on complex reasoning tasks. However, these methods rely heavily on accurate pseudo-label estimation and converge on spurious yet popular answers, thereby trapping in a dominant mode and limiting further improvements. Building on this, we propose Dual Consensus Reinforcement Learning (DCRL), a novel self-supervised training method which is capable of generating more reliable learning signals through a two-stage consensus mechanism. The model initially acts as an anchor, producing dominant responses; then it serves as an explorer, generating diverse auxiliary signals via a temporary unlearning process. The final training target is derived from the harmonic mean of these two signal sets. Notably, the process operates entirely without external models or supervision. Across eight benchmarks and diverse domains, DCRL consistently improves Pass@1 over majority vote while yielding more stable training dynamics. These results demonstrate that DCRL establishes a scalable path toward stronger reasoning without labels.
