Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces CANON, a novel label-free self-distillation method that leverages consensus from multiple solutions to provide dense, token-level supervision for large language models. By conditioning a frozen model snapshot on the majority answer and using it to guide the model's rollouts, CANON significantly enhances reasoning accuracy on mathematical and scientific tasks. The method achieves up to a 12-point improvement in pass@1 performance, outperforming existing label-free reinforcement learning approaches while using substantially less computational resources.
CANON transforms consensus into rich, token-level supervision, leading to unprecedented improvements in reasoning accuracy without the need for labeled data.
Sampling multiple solutions and returning the majority answer is among the most reliable ways to improve the reasoning accuracy of large language models without labels, and a growing family of methods converts this consensus signal into training supervision. However, existing approaches use consensus only in restricted forms: as a filter that selects solutions for fine-tuning, as a preference between answers, or as a scalar reward for reinforcement learning, discarding most of the information that the agreeing solutions contain. We present CANON (Consensus-ANchored self-distillatiON), a label-free training method that turns consensus into dense, token-level supervision. For each unlabeled prompt, CANON samples multiple solutions, extracts the majority answer, and conditions a frozen snapshot of the model on a solution that reaches it; this consensus-anchored teacher then supervises the model on its own rollouts at every token. Experiments on mathematical and scientific reasoning benchmarks show that CANON improves pass@1 by up to 12 points, outperforming label-free reinforcement learning by 6 points at a seventh of its compute and approaching a teacher conditioned on gold solutions; trained on pooled unlabeled data, it transfers to held-out benchmarks, matching training methods that use gold labels. Analysis suggests that the improvements are not pure distribution sharpening: after training, the model solves problems it previously never solved in 32 attempts, and its majority vote itself becomes more accurate.