Search papers, labs, and topics across Lattice.
This paper investigates semantic uncertainty quantification in language models, highlighting the limitations of current approaches that prioritize discrimination over calibration. They demonstrate that fixed-temperature heuristics lead to miscalibrated and poorly discriminative confidence distributions in question-answering. By optimizing a single scalar temperature at the token level, they achieve significant improvements in semantic calibration, discrimination, and entropy compared to heuristic baselines and more complex recalibration methods.
Turns out, a single, well-tuned temperature can drastically improve the reliability of language model confidence scores, outperforming more complex recalibration schemes.
Calibration is central to reliable semantic uncertainty quantification, yet prior work has largely focused on discrimination, neglecting calibration. As calibration and discrimination capture distinct aspects of uncertainty, focusing on discrimination alone yields an incomplete picture. We address this gap by systematically evaluating both aspects across a broad set of confidence measures. We show that current approaches, particularly fixed-temperature heuristics, produce systematically miscalibrated and poorly discriminative semantic confidence distributions. We demonstrate that optimising a single scalar temperature, which, we argue, provides a suitable inductive bias, is a surprisingly simple yet effective solution. Our exhaustive evaluation confirms that temperature scaling consistently improves semantic calibration, discrimination, and downstream entropy, outperforming both heuristic baselines and more expressive token-level recalibration methods on question-answering tasks.
