Search papers, labs, and topics across Lattice.
This paper critiques the conventional use of global calibration metrics for comparing large language models (LLMs), revealing that such metrics can be misleading due to their dependence on model accuracy. The authors introduce the ACE framework, which provides a more equitable evaluation by controlling for accuracy through three distinct calibration perspectives. Their findings indicate that many models previously deemed superior in calibration rankings lose their advantage when evaluated under this accuracy-controlled framework, highlighting the importance of accuracy-aware assessments in model comparison.
Calibration rankings of LLMs can flip dramatically when accounting for accuracy, challenging long-held beliefs about model performance.
Calibration evaluates whether a model confidence aligns with its empirical accuracy. Existing studies often compare the calibration of different large language models using global calibration metrics such as Expected Calibration Error and Brier Score. We begin by showing, both theoretically and empirically, that such comparisons are confounded by differences in model accuracy. For fairer cross-model comparison, we then propose ACE, an accuracy-controlled evaluation framework with three complementary views: Instance-Aligned, Distribution-Aligned, and Candidate-Aligned calibration. Across multiple benchmarks, model families, and confidence elicitation methods, we use ACE to study two practically important comparison axes, small versus large models and thinking versus non-thinking models. We find that many previously reported calibration advantages under raw global metrics weaken substantially after accuracy control. We also find that ranking reversal is frequent: models favored by raw metrics often cease to be favored once accuracy is controlled. Our results show that raw global calibration metrics are not robust for cross-model comparison, and that fair calibration comparison requires accuracy-aware evaluation.
