Search papers, labs, and topics across Lattice.
This paper conducts a comprehensive evaluation of LLM-as-a-Judge models, revealing significant discrepancies between exact-match agreement and Cohen's kappa, which leads to an overestimation of model discriminative abilities. The study analyzes 21 judges from nine providers across multiple benchmarks, uncovering critical findings such as a universal kappa deflation, substantial shifts in judge rankings, and a paradox of high reliability coexisting with notable position bias. The authors propose a Minimum Viable Validation Protocol to address these issues and improve the evaluation framework for language models.
Kappa deflation reveals that LLM-as-a-Judge models may be overstating their discriminative abilities by up to 41 percentage points.
LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test--retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency--bias paradox), and verbosity bias is small (<0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.
BAIR