Search papers, labs, and topics across Lattice.
This paper critiques the inefficiencies of fixed-size benchmarks in model evaluation, highlighting their inability to meet diverse evaluation objectives without incurring high costs or compromising reliability. To address this, the authors propose an adaptive evaluation framework that employs sequential testing and tailored stopping criteria to balance efficiency and reliability. Their framework achieves an 80% reduction in computational costs compared to traditional methods while maintaining statistical significance, demonstrating its effectiveness on the Open VLM Leaderboard.
Sequential testing can cut model evaluation costs by 80% while ensuring reliable results, challenging the status quo of fixed-size benchmarks.
The inherent rigidity of fixed-size benchmarks makes them an inefficient tool for model evaluation. Diverse evaluation objectives, including model ranking, model selection and testing throughout development, demand varying levels of statistical power. The mismatch between fixed sample sizes and these diverse needs results in either excessive computational cost or compromised reliability - a critical concern for model evaluation. To overcome these limitations, we call for adoption of sequential testing in our field. We provide an adaptive evaluation framework, that provides a principled way to navigate the trade-off between efficiency and reliability in model evaluation. Our framework combines the established statistical paradigm of sequential testing with stopping criteria tailored to common evaluation needs such as diminishing returns detection, and minimum detectable effect size. We demonstrate its ability to adaptively manage the efficiency-reliability trade-off on the Open VLM Leaderboard, including, for example, a 80% reduction in computational cost compared to fixed-size evaluation (with a 2.5-point CI width allowance) while maintaining statistical significance.
