Search papers, labs, and topics across Lattice.
The paper investigates the limitations of using Large Language Models (LLMs) as automatic evaluators in pairwise comparative judgment settings for natural language generation (NLG) assessment, highlighting inconsistencies in LLM comparison probabilities and the inadequacy of simple averaging-based aggregation. To address this, they propose BT-sigma, a judge-aware extension of the Bradley-Terry model that infers both item rankings and judge reliability from pairwise comparisons without human supervision. Experiments on benchmark NLG evaluation datasets demonstrate that BT-sigma outperforms averaging-based methods and learns a discriminator that correlates with judge consistency, effectively calibrating LLM judges in an unsupervised manner.
LLMs aren't equally reliable as NLG evaluators, but a Bradley-Terry extension called BT-sigma can learn judge reliability from pairwise comparisons alone, improving ranking accuracy without human supervision.
Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and aspects, and their judgment probabilities may be biased and inconsistent. Furthermore, human-labelled supervision for judge calibration may be unavailable. We first empirically demonstrate that inconsistencies in LLM comparison probabilities exist and show that it limits the effectiveness of direct probability-based ranking. To address this, we study the LLM-as-a-jury setting and propose BT-sigma, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone. Experiments on benchmark NLG evaluation datasets show that BT-sigma consistently outperforms averaging-based aggregation methods, and that the learned discriminator strongly correlates with independent measures of the cycle consistency of LLM judgments. Further analysis reveals that BT-sigma can be interpreted as an unsupervised calibration mechanism that improves aggregation by modelling judge reliability.
