Search papers, labs, and topics across Lattice.
This paper introduces CANON, a novel label-free self-distillation method that leverages consensus among multiple solutions to provide dense, token-level supervision for large language models. By sampling various outputs for each prompt and conditioning the model on the majority answer, CANON significantly enhances reasoning accuracy, achieving up to a 12-point improvement in pass@1 on mathematical and scientific benchmarks. The method not only outperforms traditional label-free reinforcement learning approaches but also demonstrates the ability to solve previously unsolvable problems, indicating a substantial advancement in model training efficiency and effectiveness.
CANON transforms consensus into dense supervision, boosting reasoning accuracy by up to 12 points while using a fraction of the compute of traditional methods.
Sampling multiple solutions and returning the majority answer is among the most reliable ways to improve the reasoning accuracy of large language models without labels, and a growing family of methods converts this consensus signal into training supervision. However, existing approaches use consensus only in restricted forms: as a filter that selects solutions for fine-tuning, as a preference between answers, or as a scalar reward for reinforcement learning, discarding most of the information that the agreeing solutions contain. We present CANON (Consensus-ANchored self-distillatiON), a label-free training method that turns consensus into dense, token-level supervision. For each unlabeled prompt, CANON samples multiple solutions, extracts the majority answer, and conditions a frozen snapshot of the model on a solution that reaches it; this consensus-anchored teacher then supervises the model on its own rollouts at every token. Experiments on mathematical and scientific reasoning benchmarks show that CANON improves pass@1 by up to 12 points, outperforming label-free reinforcement learning by 6 points at a seventh of its compute and approaching a teacher conditioned on gold solutions; trained on pooled unlabeled data, it transfers to held-out benchmarks, matching training methods that use gold labels. Analysis suggests that the improvements are not pure distribution sharpening: after training, the model solves problems it previously never solved in 32 attempts, and its majority vote itself becomes more accurate.
