Search papers, labs, and topics across Lattice.
The paper introduces MEDSYN, a new multilingual, multimodal benchmark designed to evaluate MLLMs on complex clinical cases involving up to 7 distinct visual clinical evidence types. The benchmark assesses MLLMs on differential diagnosis generation and final diagnosis selection, revealing a significant performance gap between the two tasks compared to human experts, indicating a failure to effectively synthesize heterogeneous clinical evidence. Through ablations and a novel "Evidence Sensitivity" metric, the authors identify over-reliance on textual evidence and a cross-modal evidence utilization gap as key contributors to this failure, demonstrating that reducing this gap improves diagnostic accuracy.
Despite matching or exceeding human expert performance on generating potential diagnoses, current MLLMs struggle to synthesize multimodal clinical evidence for final diagnosis, revealing a critical gap in their clinical reasoning abilities.
Multimodal large language models (MLLMs) have shown great potential in medical applications, yet existing benchmarks inadequately capture real-world clinical complexity. We introduce MEDSYN, a multilingual, multimodal benchmark of highly complex clinical cases with up to 7 distinct visual clinical evidence (CE) types per case. Mirroring clinical workflow, we evaluate 18 MLLMs on differential diagnosis (DDx) generation and final diagnosis (FDx) selection. While top models often match or even outperform human experts on DDx generation, all MLLMs exhibit a much larger DDx--FDx performance gap compared to expert clinicians, indicating a failure mode in synthesis of heterogeneous CE types. Ablations attribute this failure to (i) overreliance on less discriminative textual CE ($\it{e.g.}$, medical history) and (ii) a cross-modal CE utilization gap. We introduce Evidence Sensitivity to quantify the latter and show that a smaller gap correlates with higher diagnostic accuracy. Finally, we demonstrate how it can be used to guide interventions to improve model performance. We will open-source our benchmark and code.
Amazon Science