Search papers, labs, and topics across Lattice.
This study critically evaluates the assumption that self-consistency and cross-model agreement among large language models (LLMs) serve as reliable indicators of correctness in AI evaluations. Through a large-scale analysis involving 53 runners and 265,000 samples, the authors find that while agreement can be a weak predictor of accuracy, it is heavily regime-dependent, performing best with mid-tier models and poorly with high-consistency frontier models that often mislead with overconfidence. The findings highlight that self-consistency should be treated as a conditional proxy for correctness rather than a definitive confidence signal.
Agreement among LLMs can mislead evaluations, with high consistency often masking significant inaccuracies.
LLM-as-judge (Zheng et al., 2023) is increasingly the default for evaluating AI systems in enterprise pipelines, often scaled to ensembles (Verga et al., 2024) or"mixture-of-experts"(Shazeer et al., 2017) panels of judges. These systems share a key assumption: that consistency -- agreement among judges, or among a model's own samples -- indicates correctness. We show this assumption is unreliable. Agreement is not accuracy: a model can agree with itself, and different models can agree with each other, out of shared bias, a memorized heuristic, or an option-position prior rather than truth. We ask when agreement is nonetheless a usable proxy, in a large-scale cross-runner study: 53 runners drew K=50 samples for assigned overlapping cases across comparisons of model tier, prompting, and scale on GPQA Diamond and AIME -- 265,000 samples. Using majority-correctness as the deployment label and a hierarchical runner-clustered bootstrap, agreement is a positive but weak predictor (rho 0.20-0.59, all positive under item-clustered resampling) whose usefulness is regime-dependent: best for unsaturated mid-tier models and for allocating compute, and worst -- over-confident yet no more accurate -- for the most consistent frontier model (agreement>=0.8 on 77% of GPQA case-result entries, 48% of those wrong). An exploratory cross-family check on three Claude tiers shows the same frontier over-confidence, with confident errors recurring across providers above a marginal-preserving null. Self-consistency is thus a conditional proxy for correctness, not a standalone confidence score. We publicly release the de-identified per-run rows and answer distributions.
