Search papers, labs, and topics across Lattice.
This paper addresses the inefficiencies in human evaluation for model ranking in NLP by framing it as a best-arm identification problem within a multi-armed bandit framework. By adaptively sampling based on interim rankings, the authors optimize the allocation of evaluation resources, focusing on the most competitive models and thus reducing costs and time. The proposed algorithms are proven to be optimal and significantly enhance the ability to discriminate between top-performing models, aligning evaluations with large-scale competition needs.
Adaptive sampling can cut human evaluation costs while boosting the accuracy of model rankings in NLP tasks.
While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability. When identifying top-performing models, typical evaluation protocols waste effort by exhaustively evaluating all models on the entire benchmark, a safe but inefficient approach. In this work, we formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms, where pulling an arm corresponds to human-evaluating a model. By sampling adaptively based on the intermediate model rankings obtained on the samples so far, we can focus the annotation budget on the most competitive models. We prove the optimality of the proposed algorithms and show that it improves discrimination between top-performing models. This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.
ETH