Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper critiques existing LLM evaluation methods for medical SOAP note generation, arguing that they incorrectly penalize clinically valid inferences as "hallucinations" due to their reliance on lexical faithfulness. They demonstrate that many flagged hallucinations are actually legitimate clinical transformations like synonym mapping, abstraction, and diagnostic inference. By incorporating calibrated prompting and medical ontology retrieval to align evaluation with clinical reasoning, they significantly reduce the perceived hallucination rate from 35% to 9%.
Current LLM evaluation methods overestimate "hallucination" rates in medical contexts by nearly 4x because they fail to recognize valid clinical reasoning as distinct from factual error.
Evaluating large language models (LLMs) for clinical documentation tasks such as SOAP note generation remains challenging. Unlike standard summarization, these tasks require clinical abstraction, normalization of colloquial language, and medically grounded inference. However, prevailing evaluation methods including automated metrics and LLM as judge frameworks rely on lexical faithfulness, often labeling any information not explicitly present in the transcript as hallucination. We show that such approaches systematically misclassify clinically valid outputs as errors, inflating hallucination rates and distorting model assessment. Our analysis reveals that many flagged hallucinations correspond to legitimate clinical transformations, including synonym mapping, abstraction of examination findings, diagnostic inference, and guideline consistent care planning. By aligning evaluation criteria with clinical reasoning through calibrated prompting and retrieval grounded in medical ontologies we observe a significant shift in outcomes. Under a lexical evaluation regime, the mean hallucination rate is 35%, heavily penalizing valid reasoning. With inference aware evaluation, this drops to 9%, with remaining cases reflecting genuine safety concerns. These findings suggest that current evaluation practices over penalize valid clinical reasoning and may measure artifacts of evaluation design rather than true errors, underscoring the need for clinically informed evaluation in high context domains like medicine.