Search papers, labs, and topics across Lattice.
This study critically evaluates the effectiveness of self-consistency (SC) through majority voting in small instruction-tuned language models, revealing that this approach often decreases accuracy on challenging scientific questions. Analyzing performance on the GPQA Diamond benchmark, the authors found that majority voting resulted in accuracy drops for 56.6% of problems in Qwen2.5-7B and 65.7% in Llama-3-8B, with a notable disconnect between confidence levels and correctness. The research underscores the limitations of majority voting, as even advanced gating mechanisms failed to improve accuracy beyond a fixed-budget approach, highlighting a significant gap in current methodologies for handling hard science problems.
Majority voting in small LLMs can actually worsen performance on difficult science questions, contradicting its intended purpose of enhancing accuracy.
Self-consistency (SC) via majority vote is a widely used way to spend inference-time compute: sample N chains of thought, return the plurality answer. On the full GPQA Diamond benchmark (198 graduate-level science questions), majority voting reduces per-problem accuracy on a majority of problems for two instruction-tuned models from different families: 56.6% of problems for Qwen2.5-7B and 65.7% for Llama-3-8B, with Qwen the primary demonstration and Llama corroborating the direction from a near-chance baseline. The effect was pre-registered on a 151-problem confirmatory split after being observed on 47 exploratory problems, and all four confirmatory hypotheses passed. A grid oracle that routes each problem to the best N across {1, 2, 4, 8, 16, 32, 64} marks a theoretical upper bound 14 accuracy points above N = 1 for Qwen and 17 for Llama, an oracle bound requiring ground truth rather than a deployable method. No verifier-free gate reaches it: neither a plurality-agreement gate nor a token-entropy gate moves accuracy more than 0.002 from fixed-budget voting at N = 64. The mechanism is direct: confidence does not track correctness on these problems. In the highest-agreement bin the plurality answer is correct about half the time for Qwen, and for Llama that bin is less accurate than its lowest-agreement bin. We pre-register and confirm these findings on small instruction-tuned models; we do not test reasoning-native models, which we flag as the central open question.
