Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper analyzes the effectiveness of LLM-as-a-judge panels in evaluating language tasks, finding that correlated errors among models significantly reduce the panel's informational value. By applying the Kish effective sample size and a Condorcet null model to NLI datasets, the study reveals that a panel of 9 LLMs effectively provides only about 2 independent votes. Consequently, the panel's accuracy is substantially lower than what independent voting would achieve, highlighting the limitations of scaling up panels without addressing the issue of correlated errors.
LLM evaluation panels are mostly an illusion: nine judges provide only two independent opinions, because they make the same mistakes on the same examples.
LLM-as-a-judge panels aggregate votes from multiple models, with the expectation that diverse models yield more reliable evaluations. We develop a framework to measure the true informational value of such panels and quantify how far their reliability falls short of the independent-voting ideal. Testing a panel of 9 frontier LLMs from 7 model families on three natural language inference datasets (each with 100 human annotations per item), we find that the 9 judges effectively provide only about 2 independent votes'worth of information. Roughly three-quarters of the panel's nominal independence is lost because the models make the same mistakes on the same items. The consequences are stark: the panel's actual accuracy falls 8-22 percentage points short of what independent voting would achieve, and the best single judge matches or outperforms the full panel across all conditions. Neither adding more judges nor using smarter aggregation algorithms helps -- established methods close at most 11% of this gap, even with access to the correct answers. We quantify these findings using the Kish effective sample size (n_eff) and a Condorcet null model, and show the deficit is robust across prompt variants, temperatures, chain-of-thought reasoning, and a pairwise preference task (RewardBench). The bottleneck is correlated judges, not the aggregation algorithm, implying that scaling up panels cannot substitute for genuinely independent evaluation.