Search papers, labs, and topics across Lattice.
This paper addresses the problem of judge bias in LLM-as-judge evaluation, which can lead to unreliable model rankings. They introduce a variance decomposition to quantify the contributions of scenario, generation, judge, and residual variance to overall benchmark score variance. Based on this analysis, they propose CyclicJudge, a round-robin judge assignment strategy, and demonstrate both theoretically and empirically that it effectively eliminates judge bias while maintaining the cost efficiency of single-judge evaluation.
A round-robin judge assignment strategy, CyclicJudge, precisely eliminates judge bias in LLM evaluation without increasing evaluation costs, offering a practical solution to a pervasive problem.
LLM-as-judge evaluation has become standard practice for open-ended model assessment; however, judges exhibit systematic biases that cannot be eliminated by increasing the number of scenarios or generations. These biases are often similar in magnitude to the model differences that benchmarks are designed to detect, resulting in unreliable rankings when single-judge evaluations are used. This work introduces a variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components. Based on this analysis, CyclicJudge, a round-robin assignment of judges, is demonstrated to be the optimal allocation strategy. It eliminates bias precisely while requiring each judge only once per cycle, maintaining the cost of single-judge evaluation. Empirical validation on MT-Bench supports all theoretical predictions.
