Search papers, labs, and topics across Lattice.
This paper presents a controlled comparative study of distribution-based and set-based representations of epistemic uncertainty in neural networks, constructing both from the same finite collection of predictive distributions generated by a shared neural network. The study evaluates each representation using three uncertainty measures across eight benchmarks, including selective prediction and out-of-distribution detection, using six predictive models and ten independent runs per configuration. Results demonstrate that meaningful comparison is feasible and informative, providing insights into how second-order representation choices impact uncertainty-aware performance.
Second-order uncertainty representations matter: set-based and distribution-based methods, often considered incomparable, can be rigorously compared, revealing how representation choices impact uncertainty-aware performance.
Epistemic uncertainty in neural networks is commonly modeled using two second-order paradigms: distribution-based representations, which rely on posterior parameter distributions, and set-based representations based on credal sets (convex sets of probability distributions). These frameworks are often regarded as fundamentally non-comparable due to differing semantics, assumptions, and evaluation practices, leaving their relative merits unclear. Empirical comparisons are further confounded by variations in the underlying predictive models. To clarify this issue, we present a controlled comparative study enabling principled, like-for-like evaluation of the two paradigms. Both representations are constructed from the same finite collection of predictive distributions generated by a shared neural network, isolating representational effects from predictive accuracy. Our study evaluates each representation through the lens of 3 uncertainty measures across 8 benchmarks, including selective prediction and out-of-distribution detection, spanning 6 underlying predictive models and 10 independent runs per configuration. Our results show that meaningful comparison between these seemingly non-comparable frameworks is both feasible and informative, providing insights into how second-order representation choices impact practical uncertainty-aware performance.
