Search papers, labs, and topics across Lattice.
The paper identifies a "consensus trap" in label-free RL for LLMs, where models converge to confidently incorrect solutions due to collapsing output diversity when using majority-voted answers as pseudo-labels. To address this, they introduce CoVerRL, a framework that alternates a single model between generator and verifier roles, bootstrapping each other's capabilities. Experiments on mathematical reasoning benchmarks demonstrate that CoVerRL outperforms label-free baselines and significantly improves self-verification accuracy, showing genuine co-evolution of reasoning and verification.
LLMs can escape the trap of confidently wrong reasoning by co-evolving a generator and verifier from a single model, bootstrapping each other to break free from flawed consensus.
Label-free reinforcement learning enables large language models to improve reasoning capabilities without ground-truth supervision, typically by treating majority-voted answers as pseudo-labels. However, we identify a critical failure mode: as training maximizes self-consistency, output diversity collapses, causing the model to confidently reinforce systematic errors that evade detection. We term this the consensus trap. To escape it, we propose CoVerRL, a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other. Majority voting provides noisy but informative supervision for training the verifier, while the improving verifier progressively filters self-consistent errors from pseudo-labels. This co-evolution creates a virtuous cycle that maintains high reward accuracy throughout training. Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9\% on mathematical reasoning benchmarks. Moreover, self-verification accuracy improves from around 55\% to over 85\%, confirming that both capabilities genuinely co-evolve.
