Search papers, labs, and topics across Lattice.
This study introduces clinical reasoning graphs, which are structured representations derived from LLM diagnostic traces, to evaluate the reasoning consistency of large language models in clinical contexts. By analyzing 750 diagnostic traces from five LLMs across various clinical cases, the authors find that while models achieve high diagnostic accuracy, they lack consistent reasoning patterns, as evidenced by similar graph structures for both correct and incorrect diagnoses. The findings suggest that traditional accuracy metrics are insufficient for assessing the quality of diagnostic reasoning in LLMs, advocating for a process-level evaluation approach alongside accuracy assessments.
Despite achieving 60-70% diagnostic accuracy, LLMs exhibit a striking lack of consistency in clinical reasoning, revealing a gap between competence and structured reasoning.
Modern large language models (LLMs) reach 60-70% diagnostic accuracy on complex clinical case benchmarks, but accuracy alone cannot distinguish stable clinically-grounded reasoning from pattern matching. We introduce clinical reasoning graphs, structured graph representations extracted from free-text LLM diagnostic traces using a domain-grounded ontology with 5 node types and 7 edge types. We apply this pipeline to 750 traces from five LLMs across 50 New England Journal of Medicine Clinicopathological Conference cases and three prompt conditions, and test whether diagnostic traces show stable structured reasoning patterns, or diagnostic schemas, for clinically similar cases. We operationalize this as higher graph similarity among clinically similar cases than among clinically dissimilar ones. Across 15 model-condition comparisons, within-cluster and between-cluster composite similarity are nearly equal, and no comparison survives multiple-testing correction; a component-level analysis finds any residual content signal far below schema scale. Graph similarity is also nearly identical for pairs of models that are both correct (0.488) and both incorrect (0.484), suggesting that graph structure captures a dimension not reflected in diagnostic accuracy. Structured reflection prompting increases explicit discriminating-feature analysis within traces (+33%) but does not increase cross-case consistency. These results show diagnostic competence without schema-scale reasoning consistency, and indicate that final-answer accuracy should be complemented by process-level evaluation. We release the ontology, extraction pipeline, validation protocol, and the extracted reasoning graphs and similarity artifacts as resources for structured evaluation of LLM clinical reasoning.
