Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of pairwise comparisons, particularly using Elo ranking, in assessing generative models against ground-truth accuracy benchmarks. The authors demonstrate that Elo rankings achieve a Spearman correlation above 0.9 with accuracy rankings, indicating a strong alignment between subjective evaluations and objective performance metrics. Notably, the influence of stylistic cues and judge biases on rankings is minimal, with the phenomenon of answer repetition emerging as a significant factor in judge preferences.
Elo rankings can accurately reflect model performance, achieving over 90% correlation with ground-truth accuracy, even amidst stylistic biases.
Pairwise comparisons combined with aggregation methods like Elo have become central to evaluating generative models, yet concerns remain that they reward superficial stylistic cues or display judge biases. In a more positive turn, we show that model rankings from pairwise comparisons strongly agree with ground-truth-based accuracy rankings when such ground truth is available for comparison. By converting five well-known benchmarks into free-form generative evaluations, we find that Elo rankings achieve a Spearman correlation above 0.9 with accuracy rankings and substantially outperform direct evaluation when the judge is weak. Furthermore, style and judge bias have only minor effects on model rankings, despite most judgments occurring on pairs where both candidate answers are correct (or incorrect). On such pairs, we find that repetition after the final answer (echo) is a causal driver of judge preference.
