Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper analyzes the statistical resolution of pairwise LLM comparisons on public leaderboards, finding that a significant fraction of reported rankings (11/40 on Open LLM Leaderboard v1 and 4/9 on MMLU-Pro) are statistically unresolved at conventional power levels. They introduce a resolution ratio diagnostic, q = N/N*, to quantify this issue and demonstrate that common shortcuts for estimating required sample sizes in paired comparisons, like the Cohen-h-plus-(1-rho) method, can underestimate the necessary sample size by a factor of two. Their analysis holds even after accounting for multiplicity correction and sequential testing.
Many LLM leaderboard rankings are statistically meaningless: up to 27.5% of pairwise comparisons are unresolved even with standard statistical power.
Across two public LLM leaderboards, many displayed pairwise rankings do not meet a conventional paired-test resolution target under the actual paired evaluation design: 11 of 40 Open LLM Leaderboard v1 pairwise comparisons and 4 of 9 MMLU-Pro top-10 adjacent-rank pairs are unresolved at (alpha, 1-beta) = (0.05, 0.8). The MMLU-Pro count rises to 6/9 under real subject-level clustering and stays at 5-6 out of 9 in 99.9% of category-bootstrap resamples. We frame paired LLM evaluation as a hypothesis-testing problem, invert level-alpha, power-(1-beta) tests, and report a per-pair resolution ratio q = N/N* as the primary diagnostic. A sharp small-effect expansion with an explicit second-order constant shows that the widely-used unpaired Cohen-h-plus-(1-rho) shortcut deviates from the correct N* by approximately a factor of two in the close-comparison regime, a deficit that three of five off-the-shelf calculators(Cohen 1988, G*Power, R pwr) silently inherit when the user post-multiplies their per-arm output by (1-rho). The unresolved-pair pattern remains under multiplicity correction and anytime-valid sequential testing.