Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper reframes efficient LLM benchmarking as a feature selection and multiple regression problem, improving prediction accuracy and ranking correlation. They show that kernel ridge regression outperforms existing prediction methods, and that an information-theoretic feature selection algorithm (mRMR) identifies more informative question subsets. Empirically, their approach achieves lower prediction errors and higher ranking correlations across various benchmarks, while also being computationally faster and more stable than existing methods.
Forget complex efficient benchmarking schemes – simple feature selection with kernel ridge regression and mRMR can drastically improve prediction accuracy and stability while slashing computation time.
Efficient benchmarking techniques aim to lower the computational cost of evaluating LLMs by predicting full benchmark scores using only a subset of a benchmark's questions. By reframing this problem as an instance of multiple regression with feature selection, we find that existing efficient benchmarking methods can be greatly improved by simply using kernel ridge regression at the prediction stage. Additionally, using an information-theoretic feature-selection algorithm called minimum redundancy maximum relevance (mRMR), we can further improve upon these methods by selecting question subsets that will be maximally useful for prediction. Except in very data-poor settings, these approaches consistently achieve smaller prediction errors (in both MAE and RMSE), and greater ranking correlation between predicted and true scores (in both Spearman $ρ$ and Kendall $τ$) across a range of benchmarks using both binary and continuous metrics. Furthermore, mRMR subsampling is much faster than competitor methods (which often involve fitting probabilistic models or running clustering algorithms), and is more likely to select the same questions under different random seeds or training data splits. Tutorial code can be found at https://github.com/sambowyer/mrmr_eval .