Search papers, labs, and topics across Lattice.
This paper introduces MTDiag, a multi-turn diagnostic dataset designed to evaluate the performance of Large Language Models (LLMs) in dynamic clinical settings, addressing the limitations of static QA benchmarks. By integrating diverse clinical sources and normalizing cases into a comprehensive schema, the dataset facilitates the assessment of LLMs' capabilities in handling both common and rare medical conditions during interactive dialogues. The authors also propose new clinical knowledge-grounded metrics that extend evaluation criteria beyond mere diagnostic accuracy, emphasizing the importance of contextual understanding in multi-turn interactions.
LLMs struggle significantly in multi-turn clinical dialogues, with MTDiag revealing their diagnostic capabilities in real-world scenarios where static benchmarks fall short.
Clinical diagnosis is fundamentally interactive and incremental, yet the dominant paradigm for evaluating Large Language Models (LLMs) in medicine remains static QA benchmarks or template-based dialogues. These benchmarks say little about whether a model can serve as a diagnostic agent in a dynamic clinical encounter, with LLMs showing significant accuracy and reliability degradation in multi-turn settings. To address this issue, we present MTDiag, a large multi-turn diagnostic dialogue dataset constructed from three heterogeneous sources: DDXPlus, MIMIC-IV, and published case reports (AJCR), covering common ED presentations as well as long-tail rare and atypical conditions. All cases are normalized into a canonical schema anchored in the most comprehensive and widely-adopted medical knowledge bases (UMLS concept identifiers, with ICD-10 diagnosis codes). We release the schema, a UserLM-8B-based utterance-generation pipeline, and the physician-validated dataset that converts structured clinical evidence into natural-language utterances. Importantly, we introduce and motivate clinical knowledge-grounded metrics for evaluating LLMs as diagnostic agents, beyond diagnostic accuracy, for the task of multi-turn differential diagnosis.
