Search papers, labs, and topics across Lattice.
This paper critiques traditional metrics for evaluating uncertainty quantification (UQ) in machine learning, revealing that they often fail to align with the actual utility in downstream decision-making. By introducing the decision-alignment framework, the authors identify misalignments in widely used metrics and propose prior-weighted utility metrics that better reflect decision relevance. Their experiments demonstrate that these new metrics consistently outperform conventional ones in aligning with realized decision utility, highlighting significant flaws in current UQ evaluation practices.
Many popular uncertainty metrics mislead decision-making, but new decision-aligned metrics show a striking improvement in utility alignment.
Uncertainty estimates in machine learning are typically evaluated using generic metrics such as the negative log-likelihood and expected calibration error, yet good performance on such metrics does not necessarily imply high utility in downstream decisions. We introduce decision-alignment, a criterion that reveals which evaluation metrics meaningfully align with downstream utilities. Applying this framework, we show that many widely used uncertainty metrics are either misaligned with common decision problems or encode pathological prior beliefs about the downstream task. We then propose prior-weighted utility metrics, a special class of proper scoring rules that provides decision-aligned uncertainty evaluation. Across benchmark experiments and real-world case studies, our metrics consistently align with realized decision utility, while conventional metrics do not. Our results surface flaws in the current UQ evaluation protocol and offer a principled extension of existing metrics toward decision-relevant UQ evaluation.
