Search papers, labs, and topics across Lattice.
This study investigates the reliability of large language models (LLMs) as judges by analyzing how changes in evaluators affect scoring, even when candidate responses remain constant. Through comparisons of different model upgrades across four judgment datasets, the authors find that not all upgrades yield equivalent improvements; specifically, only the upgrade from Qwen3 1.7B to 4B shows significant reliability gains, while MiniMax upgrades do not. The research highlights persistent biases in LLM judgments and emphasizes the need for comprehensive reporting practices to enhance measurement validity and transparency in LLM evaluations.
Judge upgrades in LLMs aren't interchangeable鈥攐nly specific parameter increases yield reliable improvements in evaluation consistency.
An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed. We treat this evaluator-replacement ambiguity as a measurement-validity problem. Across four judgment datasets, we compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2-M2.7 released APIs. The main pattern is that judge upgrades are not interchangeable: only Qwen3 1.7B to 4B gives a robust adjacent gain, while MiniMax adjacent releases do not. Stronger judges reduce but do not remove position and verbosity bias. Repeated-sample juries add little when errors are correlated. Structured debate can move decisions substantially, but without parser and fallback logs those shifts cannot be attributed to deliberation. We argue that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails.
