Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the concept of epistemic calibration, which evaluates whether the reported epistemic uncertainty of second-order classification models accurately reflects the variability of model predictions relative to the true outcomes. The authors demonstrate that epistemic calibration is a more robust measure than classical calibration, revealing failure modes that traditional metrics overlook. Through the development of the Expected Epistemic Calibration Error (EECE), they provide a consistent estimator for the True Epistemic Calibration Error (TECE), supported by experiments showing significant discrepancies in uncertainty quantification methods despite comparable predictive accuracy.
Epistemic calibration reveals hidden vulnerabilities in uncertainty estimates that classical calibration fails to detect, impacting model trustworthiness in critical applications.
Uncertainty estimation is critical for deploying machine learning models in high-stakes settings. However, classical calibration only assesses the reliability of predicted probabilities and does not evaluate whether epistemic uncertainty estimates are themselves trustworthy. This limitation is particularly relevant for second-order classification models. We introduce epistemic calibration, a principled criterion that measures whether reported epistemic uncertainty faithfully reflects the dispersion of model predictions around the ground truth. We show that epistemic calibration is a strictly stronger notion than classical calibration and captures failure modes invisible to standard metrics. We relate this work to the existing literature through an impossibility theorem that holds under the epistemic calibration hypothesis. To operationalize this concept, we propose the Expected Epistemic Calibration Error (EECE), which we prove to be a consistent estimator of a True Epistemic Calibration Error (TECE). Experiments across a broad range of uncertainty quantification methods show that epistemic calibration is a coherent and meaningful criterion and reveal substantial differences across methods, despite similar predictive performance.