Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
To isolate whether clinical decisions stem from sound reasoning or lucky guesses, the authors evaluate nine LLMs on MedEVM, a 1,050-case benchmark requiring models to dynamically balance evidence acquisition against diagnostic commitment across sequential turns. They identify pervasive Evidence-Value Misalignment (EVM), showing that models routinely suffer from premature submission, sensitivity to evidence ordering, and paradoxically worse evidence calibration when using extended reasoning modes. Gating decisions with EVD-Harness—a framework that decouples diagnosis generation from submission via contrastive witness verification—mitigates these failures and improves diagnostic accuracy by 12.0 to 51.1 percentage points across five models.
Extended reasoning modes paradoxically degrade an LLM's ability to assess evidence sufficiency in dynamic diagnosis, driving models to prematurely lock into diagnoses that succeed only by chance.
A correct diagnosis reached from insufficient or misleading evidence can pose a clinical hazard, yet outcome-based accuracy may reward such lucky guesses. We call this mismatch between diagnostic decisions and the value of available evidence Evidence-Value Misalignment (EVM). To disentangle evidential grounding independently from diagnostic accuracy, we introduce MedEVM, a dynamic benchmarking environment comprising 1,050 cases across 24 disease systems. Observations arrive turn by turn, requiring models to continuously calibrate its decision by deciding whether to wait for more evidence or submit a diagnosis. Across 9 LLMs, four interesting patterns are observed. (1) Miscalibrated evidence tracking. Making a diagnosis often fails to calibrate evidence sufficiency, even in more capable models, and even worsens in reasoning mode. (2) Misaligned diagnosis submission. Confidence in the correct diagnosis often fails to ensure timely submission despite sufficient evidence. (3) Evidence order matters. Reordering the same evidence changes diagnoses even when model confidence remains similar. (4) Misleading evidence remains influential. Added misleading evidence redirects diagnoses even after prior evidence becomes sufficient. We further verify that EVM predicts errors and that preventing premature submission improves accuracy. These findings motivate Evidence-Verified Diagnosis Harness (EVD-Harness). It decouples diagnosis generation from submission through an offline Contrastive Diagnostic Wiki and three online control stages, namely observation management, proposal and witness verification, and diagnosis submission control. Across five LLMs, EVD-Harness improves accuracy by 12.0--51.1 percentage points while mitigating EVM-related failures. Our results demonstrate that verifying evidential support before submission can make diagnostic decisions more reliable.