Search papers, labs, and topics across Lattice.
This paper addresses the challenge of bias in large language model (LLM) judges by framing the ranking of outputs as Bayesian inference that incorporates judge-specific biases, such as verbosity and position effects. By introducing a top-k-aware active acquisition rule, the authors demonstrate that their approach significantly improves the identification of the top-k items while requiring fewer comparisons than traditional methods. The results show that while lower-tier judges exhibit strong verbosity biases that can mislead rankings, the proposed model effectively corrects for these biases, enhancing recall from approximately 0.5-0.6 to 0.84-1.0.
Bias in LLM judges can be corrected to improve ranking accuracy, lifting recall rates significantly in noisy environments.
Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise -- to rank responses, select models, or triage papers. Yet LLM judges are both noisy and systematically biased: they favor verbose or well-formatted answers and exhibit position effects, so simply aggregating their votes recovers a ranking of presentation, not of true quality. We study the practical goal of identifying the \topk{} items under a fixed comparison budget, and make two contributions. First, we cast judging as Bayesian inference over latent quality with explicit, judge-specific bias covariates (verbosity, position), regularized by a shrinkage prior so that the data decide which biases a given judge actually exhibits. Second, we introduce a \topk-aware active acquisition rule that chooses the next comparison to maximally reduce uncertainty about \topk{} \emph{membership}, rather than about the full ranking. On a controlled benchmark with known ground-truth quality, judged by sixteen real LLMs spanning open and proprietary families (Llama, Qwen, Phi-4, GPT-4o-mini/5.1/5.5, Gemini, DeepSeek, and Claude Haiku/Sonnet/Opus), naive aggregation plateaus at a wrong \topk{} on biased judges regardless of budget, while our bias-aware model recovers it; \topk-aware acquisition reaches this ceiling with far fewer comparisons than round-robin or a global-uncertainty (D-optimal) rule. Bias is real but heterogeneous and capability-dependent: cheap and mid-tier judges carry a strong verbosity bias that our model corrects (lifting recall from $\sim$$0.5$--$0.6$ to $0.84$--$1.0$), whereas the frontier judges we tested show little bias and already rank accurately, so bias-aware modeling changes little there.
Tsinghua AI