Search papers, labs, and topics across Lattice.
This paper introduces a unified evaluation framework for assessing classifier reliability by jointly considering out-of-distribution (OOD) detection and in-distribution (ID) failure prediction. They propose new metrics, DS-F1 and DS-AURC, based on double scoring functions to quantify this joint reliability. Experiments on OpenOOD demonstrate that double scoring functions improve reliability compared to single scoring approaches, and the authors also introduce SURE+, a method that further enhances reliability across various scenarios.
Classifiers can be made substantially more reliable by jointly considering out-of-distribution detection and in-distribution failure prediction, a connection often overlooked in prior work.
Building reliable classifiers is a fundamental challenge for deploying machine learning in real-world applications. A reliable system should not only detect out-of-distribution (OOD) inputs but also anticipate in-distribution (ID) errors by assigning low confidence to potentially misclassified samples. Yet, most prior work treats OOD detection and failure prediction as separated problems, overlooking their closed connection. We argue that reliability requires evaluating them jointly. To this end, we propose a unified evaluation framework that integrates OOD detection and failure prediction, quantified by our new metrics DS-F1 and DS-AURC, where DS denotes double scoring functions. Experiments on the OpenOOD benchmark show that double scoring functions yield classifiers that are substantially more reliable than traditional single scoring approaches. Our analysis further reveals that OOD-based approaches provide notable gains under simple or far-OOD shifts, but only marginal benefits under more challenging near-OOD conditions. Beyond evaluation, we extend the reliable classifier SURE and introduce SURE+, a new approach that significantly improves reliability across diverse scenarios. Together, our framework, metrics, and method establish a new benchmark for trustworthy classification and offer practical guidance for deploying robust models in real-world settings. The source code is publicly available at https://github.com/Intellindust-AI-Lab/SUREPlus.
