Search papers, labs, and topics across Lattice.
CMU ML
MIT CSAIL
NUS
Stanford HAIAffiliation:CambridgeCornellDepartment of Pathology and Laboratory MedicineDukeHarvardNTURIKENUBUNIGEUPennUTokyoYaleJul 28, 2026arXiv:2607.25933This study introduces ClinMM-Bench, the largest multi-turn multimodal clinical diagnostic evaluation benchmark, comprising 1,089 complex clinical cases and 3,760 medical images across eight specialties. By evaluating 15 multimodal large language models (MLLMs) through a two-level framework assessing both diagnostic accuracy and reasoning quality, the research reveals that while proprietary models excel in diagnostic accuracy, their ability to generate reliable reasoning remains significantly constrained. The analysis uncovers five critical failure modes, highlighting the need for improved synthesis and reasoning capabilities in MLLMs for clinical applications.
Proprietary MLLMs may achieve high diagnostic accuracy, but they still struggle with reliable clinical reasoning, revealing significant gaps in their practical utility.
Clinical diagnostic evaluation should not only assess whether models can provide correct diagnoses, but also reflect the realities of clinical practice, including progressive disclosure of multimodal information, dynamic updating of diagnostic hypotheses, and continuous refinement of clinical reasoning. However, existing evaluations of multimodal large language models (MLLMs) typically rely on single-turn or isolated tasks, making it difficult to fully capture the complexity of real-world clinical diagnosis. To bridge this gap, we developed ClinMM-Bench, the largest multi-turn multimodal clinical diagnostic evaluation benchmark to date. ClinMM-Bench contains 1,089 challenging real-world clinical cases and 3,760 medical images across eight specialties. We systematically evaluated 15 representative MLLMs using a two-level evaluation framework that assessed both diagnostic accuracy and diagnostic reasoning quality. Results showed that proprietary models achieved the highest overall diagnostic accuracy, but the proportion of completely correct diagnoses remained limited across all models. In terms of diagnostic reasoning quality, current models can identify plausible diagnostic directions but still have considerable limitations in generating reliable diagnostic reasoning. Error analysis further identified five representative failure modes: information synthesis failure, knowledge mapping error, perception error, premature closure, and visual hallucination.
