Search papers, labs, and topics across Lattice.
This paper introduces a source-grounded framework for constructing and evaluating progressive multimodal diagnostic dialogues based on clinical case reports, addressing the limitations of existing benchmarks that fail to capture interactive diagnostic processes. The framework was validated on 24 internal medicine case reports, achieving a remarkable diagnosis F1 score of 0.99 and a high reasoning-quality score of 4.79 out of 5. In contrast, evaluations of two advanced multimodal language models revealed significantly lower reasoning and diagnostic performance, underscoring the gap between fluent responses and evidence-grounded clinical reasoning.
Fluent responses from multimodal language models can mislead, as they often lack the evidence-grounded clinical reasoning necessary for accurate diagnosis.
Clinical diagnosis requires progressive integration of patient history, physical examination, laboratory findings, medical images, and diagnostic-informative tests. However, most multimodal medical benchmarks evaluate fixed inputs or endpoint answers, while fully interactive diagnostic agents conflate evidence selection with evidence interpretation. We present a source-grounded framework to construct progressive multimodal diagnostic dialogues from case reports and an evaluation strategy for assessing MLLMs on final diagnosis, diagnostic reasoning, and image-finding interpretation. Evaluation on 24 internal medicine case reports showed that our framework can accurately convert case reports into reference dialogues, achieving a diagnosis F1 of 0.99 and a reasoning-quality score of 4.79 out of 5. Evaluation on two frontier MLLMs (o4-mini and Claude Haiku 4.5) achieved reasoning-quality scores of 2.75 and 2.50, respectively, with substantially lower diagnosis, reasoning, and image-finding F1 scores. The results demonstrate that fluent responses do not necessarily reflect evidence-grounded clinical reasoning and highlight the utility of the proposed framework for evaluating multimodal diagnostic reasoning.
