Search papers, labs, and topics across Lattice.
This paper investigates the impact of anchor selection on the reliability of "LLM-as-a-judge" benchmarks, which are used to evaluate open-ended generation. Through a systematic evaluation of 22 different anchors on the Arena-Hard-v2.0 dataset, the authors demonstrate that anchor choice significantly affects correlation with human rankings, with extreme (best or worst) anchors performing poorly. They quantify the effect size of anchor selection and provide recommendations for selecting informative anchors and determining sufficient benchmark sizes for reliable evaluation.
Using a top or bottom-performing LLM as an anchor in "LLM-as-a-judge" benchmarks can dramatically skew results, making the choice of a mediocre anchor key to reliable evaluation.
The ``LLM-as-a-judge''paradigm has become a standard method for evaluating open-ended generation. To address the quadratic scalability costs of pairwise comparisons, popular benchmarks like Arena-Hard and AlpacaEval compare all models against a single anchor. However, despite its widespread use, the impact of anchor selection on the reliability of the results remains largely unexplored. In this work, we systematically investigate the effect of anchor selection by evaluating 22 different anchors on the Arena-Hard-v2.0 dataset. We find that the choice of anchor is critical: a poor anchor can dramatically reduce correlation with human rankings. We identify that common anchor choices (best-performing and worst-performing models) make poor anchors. Because these extreme anchors are consistently better or worse than all other models, they are seldom indicative of the relative ranking of the models. We further quantify the effect size of anchor selection, showing it is comparable to the selection of a judge model. We conclude with actionable recommendations. First, we conduct a power analysis, and compute sufficient benchmark sizes for anchor-based evaluation, finding that standard benchmark sizes are insufficient for pairwise evaluation and fail to distinguish between competitive models reliably. Second, we provide guidelines for selecting informative anchors to ensure reliable and efficient evaluation practices.
MIT CSAIL