Search papers, labs, and topics across Lattice.
This paper introduces a framework for efficient and extensible LLM benchmarking based on multidimensional Item Response Theory (IRT). The method uses fixed anchor items to calibrate new benchmarks to an existing evaluation suite, enabling comparison of model performance across different evaluation periods even when models are evaluated on different datasets. Experiments on over 400 models demonstrate that the framework accurately predicts full-evaluation performance (within 2-3 percentage points) using only 100 anchor questions per dataset, maintaining high rank correlation (Spearman ρ ≥ 0.9).
Stop re-running full benchmarks: Calibrate new LLM datasets against existing suites with just 100 "anchor" questions and still get highly accurate performance predictions.
The rapid release of both language models and benchmarks makes it increasingly costly to evaluate every model on every dataset. In practice, models are often evaluated on different samples, making scores difficult to compare across studies. To address this, we propose a framework based on multidimensional Item Response Theory (IRT) that uses anchor items to calibrate new benchmarks to the evaluation suite while holding previously calibrated item parameters fixed. Our approach supports a realistic evaluation setting in which datasets are introduced over time and models are evaluated only on the datasets available at the time of evaluation, while a fixed anchor set for each dataset is used so that results from different evaluation periods can be compared directly. In large-scale experiments on more than $400$ models, our framework predicts full-evaluation performance within 2-3 percentage points using only $100$ anchor questions per dataset, with Spearman $ρ\geq 0.9$ for ranking preservation, showing that it is possible to extend benchmark suites over time while preserving score comparability, at a constant evaluation cost per new dataset. Code available at https://github.com/eliyahabba/growing-pains
AI2