Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper addresses the problem of efficiently allocating a fixed computational budget when using LLMs-as-judges to evaluate prompt-response pairs, where multiple queries are needed due to stochasticity. They propose a variance-adaptive query allocation method based on multi-armed bandit theory and concentration inequalities, dynamically allocating more queries to pairs with higher score variance. The proposed algorithm achieves a near-optimal worst-case score-estimation error of $\tilde{O}\left(\sqrt{\frac{\sum_{i=1}^K σ_i^2}{B}}\right)$, and experiments on Summarize-From-Feedback and HelpSteer2 datasets demonstrate significant improvements over uniform allocation.
Stop wasting compute on LLM evals: a variance-adaptive querying strategy slashes estimation error by focusing on the most uncertain prompt-response pairs.
LLM-as-a-judge has emerged as a cornerstone technique for evaluating large language models by leveraging LLM reasoning to score prompt-response pairs. Since LLM judgments are stochastic, practitioners commonly query each pair multiple times to estimate mean scores accurately. This raises a critical challenge: given a fixed computational budget $B$, how to optimally allocate queries across $K$ prompt-response pairs to minimize estimation error? % We present a principled variance-adaptive approach leveraging multi-armed bandit theory and concentration inequalities. Our method dynamically allocates queries based on estimated score variances, concentrating resources where uncertainty is highest. Further, our algorithm is shown to achieve a worst-case score-estimation error of $\tilde{O}\left(\sqrt{\frac{\sum_{i=1}^K σ_i^2}{B}}\right)$, $σ_i^2$ being the unknown score variance for pair $i \in [K]$ with near-optimal budget allocation. % Experiments on \emph{Summarize-From-Feedback} and \emph{HelpSteer2} demonstrate that our method significantly outperforms uniform allocation, reducing worst-case estimation error while maintaining identical budgets. Our work establishes a theoretical foundation for efficient LLM evaluation with practical implications for AI safety, model alignment, and automated assessment at scale.