Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Laplace-PSN-IRT, an enhancement to the existing PSN-IRT framework that incorporates a last-layer Laplace approximation for uncertainty quantification in neural Item Response Theory models applied to large language model benchmarks. By enabling approximate Bayesian posterior inference, the method allows for calibrated uncertainty estimates regarding model ability and item difficulty without the need for retraining. The authors demonstrate that many pairwise model comparisons are statistically indistinguishable despite differing point estimates, highlighting the importance of uncertainty quantification in evaluating model performance.
Most model comparisons on LLM benchmarks are statistically indistinguishable, revealing the limitations of relying solely on point estimates for evaluation.
Item Response Theory (IRT) has recently been proposed as a framework for evaluating large language model (LLM) benchmarks by separating a model's latent ability from the properties of individual benchmark items. Existing neural IRT approaches, including PSN-IRT, estimate these quantities using point estimates, limiting uncertainty quantification and downstream statistical inference. We introduce Laplace-PSN-IRT, a post-hoc last-layer Laplace approximation that augments a trained PSN-IRT model with approximate Bayesian posterior inference, recovering calibrated uncertainty over model ability and item difficulty without retraining. The resulting posterior enables credible intervals, probabilistic comparisons between models, and propagation of parameter uncertainty into Fisher-information-based item selection. We show that most pairwise comparisons among 12 models on a standard LLM benchmark leaderboard are not statistically distinguishable despite differing point-estimate ranks. We further show that point-estimate Fisher information can become nearly zero for many benchmark items because it is evaluated at a single reference ability, whereas posterior-expected Fisher information remains substantially more stable across the ability range. Finally, posterior-expected Fisher information more accurately recovers full-benchmark ability rankings from small benchmark subsets in most experimental settings while matching point-estimate performance for the smallest subsets. We validate the calibration of the approximate posterior using held-out predictive coverage and find that modeling item difficulty as random while treating item discrimination as fixed produces well-calibrated uncertainty in this architecture.