Search papers, labs, and topics across Lattice.
This paper introduces CLExEval, a human-in-the-loop framework designed to evaluate the clinical reasoning of large language models (LLMs) by employing expert physician annotations and clinical reasoning traces. The study reveals significant evaluation illusions, such as a drastic drop in diagnostic accuracy under information scarcity and a mismatch between reasoning and final outputs, highlighting critical failure patterns in LLMs. The findings underscore the necessity of expert validation in automated clinical evaluations to avoid overestimating the reliability of LLM-generated diagnoses.
LLMs can exhibit a staggering 68.6% mismatch between their reasoning and final diagnostic outputs, raising serious concerns about their reliability in clinical settings.
Large Language Models (LLMs) achieve strong results on many medical benchmarks, but their clinical reasoning remains difficult to evaluate reliably. A central risk is an evaluation illusion: fluent and well-structured explanations can appear clinically convincing even when the final diagnosis is incorrect. We introduce CLExEval, a human-in-the-loop framework for evaluating LLM clinical reasoning under progressive information masking. CLExEval combines 5,600 expert-physician annotations with 200 clinical reasoning traces derived from 40 rare diagnostic cases. Our analysis identifies three recurring failure patterns: (i) verbosity bias, where GPT-4o-mini's diagnostic accuracy drops from 95.0% to 32.5% under information scarcity; (ii) a hidden knowledge paradox, where a specialist model reaches 92.5% maximum diagnostic potential but fails to retrieve that knowledge reliably in verbose contexts; and (iii) a 68.6% reasoning-to-output mismatch, where correct diagnoses appear in reasoning traces but are not reflected in final answers. We further evaluate the LLM-as-a-Judge paradigm on a human-verified failure set (n = 142). GPT-4o-mini approved 47.9% of clinically incorrect outputs, while HuatuoGPT-o1 approved all validly scored failures and showed a positive self-preference bias. These results suggest that standalone automated clinical evaluations can substantially overestimate clinical reliability without expert-grounded validation.
