Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel uncertainty estimation method for medical AI that leverages disagreement among expert responses to generate training targets. The approach decomposes uncertainty into aleatoric and epistemic components using a two-ensemble method, trained jointly with standard data labels. Experiments across image classification, segmentation, and question answering tasks demonstrate significant improvements in uncertainty estimation quality (9-50%) compared to existing methods.
Expert disagreement, often seen as a problem, becomes a powerful signal for training more reliable and risk-aware medical AI.
Artificial intelligence (AI) systems accelerate medical workflows and improve diagnostic accuracy in healthcare, serving as second-opinion systems. However, the unpredictability of AI errors poses a significant challenge, particularly in healthcare contexts, where mistakes can have severe consequences. A widely adopted safeguard is to pair predictions with uncertainty estimation, enabling human experts to focus on high-risk cases while streamlining routine verification. Current uncertainty estimation methods, however, remain limited, particularly in quantifying aleatoric uncertainty, which arises from data ambiguity and noise. To address this, we propose a novel approach that leverages disagreement in expert responses to generate targets for training machine learning models. These targets are used in conjunction with standard data labels to estimate two components of uncertainty separately, as given by the law of total variance, via a two-ensemble approach, as well as its lightweight variant. We validate our method on binary image classification, binary and multi-class image segmentation, and multiple-choice question answering. Our experiments demonstrate that incorporating expert knowledge can enhance uncertainty estimation quality by $9\%$ to $50\%$ depending on the task, making this source of information invaluable for the construction of risk-aware AI systems in healthcare applications.