Search papers, labs, and topics across Lattice.
This study evaluates the ability of large language models (LLMs) to preserve diagnostic uncertainty in clinical text, a critical aspect for guiding medical decision-making. By constructing a benchmark of 1,200 clinical documents with 9,184 uncertainty annotations, the authors assessed three LLMs and found that they often fail to maintain the original uncertainty cues, achieving preservation rates below 50%. The findings highlight a significant oversight in current evaluation metrics, underscoring the potential risks of deploying LLMs in clinical settings without addressing this issue.
LLMs fail to preserve diagnostic uncertainty in clinical text more than half the time, risking miscommunication in patient care.
Large language models (LLMs) are increasingly used for clinical text tasks such as summarization and revision. While most studies evaluate the fluency and coherence of LLM-generated text, whether LLMs correctly preserve diagnostic uncertainty remains underexplored. In clinical practice, phrases such as ``possible pneumonia'' communicate the strength of available evidence and directly guide decisions about follow-up testing and treatment. Altering these uncertainty expressions can change the clinical meaning entirely. In this paper, we systematically evaluated this problem in two steps. First, we constructed a benchmark of 1,200 clinical documents with 9,184 uncertainty annotations across five levels. Second, we evaluated three LLMs on this benchmark. Our results show that (1) LLMs preserve the original uncertainty cues poorly, often less than half the time; (2) LLMs struggle with nuanced distinctions between adjacent levels. This work reveals a failure mode not captured by standard evaluation metrics and provides implications for the safe deployment of LLMs in clinical workflows.
