Search papers, labs, and topics across Lattice.
This paper conducts a systematic behavioral analysis of Large Language Models (LLMs) in the context of clinical information uncertainty, revealing that while model accuracy declines with increasing uncertainty, the models' confidence levels do not align with this degradation. Using a novel evaluation framework based on the MedMCQA dataset, the authors simulate ambiguous clinical contexts and assess model responses when correct answers are deliberately omitted. The findings highlight a critical failure mode where LLMs exhibit unsafe confident errors, underscoring the need for improved uncertainty-aware evaluation methods before deploying these models in clinical settings.
LLMs may exude confidence even when they lack sufficient information, leading to dangerous misguidance in clinical decision-making.
Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks. However, their reliability under uncertainty remains poorly understood which raises critical concerns for deployment in high-stakes clinical settings. In such environments, incorrect predictions are inherently risky, but confident incorrect predictions can be particularly harmful as they may mislead clinical decision-making. In this paper, we conduct a systematic behavioral analysis of LLMs under clinical information uncertainty. We propose an evaluation framework based on the MedMCQA dataset consisting of two complementary uncertainty settings. First, we introduce linguistic uncertainty cues through prompt modifications to simulate ambiguous clinical contexts. Second, we construct an answer removal setting, wherein the correct option is deliberately excluded mandating the model to recognize insufficient information and abstain. We analyze both model accuracy and confidence behavior using multiple calibration metrics including calibration gap, Expected Calibration Error (ECE), and Unsafe Confident Error Rate (UCER) across 500 medical questions. Our results reveal a consistent failure mode, i.e., although accuracy degrades under increasing uncertainty, model confidence remains misaligned with accuracy. This leads to a substantial increase in unsafe confident errors, indicating that model confidence remains largely insensitive to clinically meaningful information loss. Furthermore, we observe significant variation across models in their ability to abstain when the correct answer is unavailable, with some models persistently producing high confidence hallucinated answers. These findings expose critical limitations in the epistemic reliability of current LLMs and highlight the need for uncertainty aware evaluation methods prior to their deployment in clinical workflows.
