Search papers, labs, and topics across Lattice.
This study introduces LingxiDiagBench, a comprehensive multi-agent framework designed to benchmark large language models (LLMs) in the context of Chinese psychiatric consultation and diagnosis. By leveraging a dataset of 16,000 EMR-aligned synthetic dialogues, the research highlights significant performance gaps in LLMs, particularly in recognizing comorbidities and conducting dynamic consultations, where accuracy dropped to 43% and 28.5%, respectively. The findings underscore the limitations of LLMs in psychiatric settings, revealing that improved questioning strategies do not necessarily translate to better diagnostic outcomes.
LLMs excel in binary classifications but struggle dramatically with complex psychiatric diagnoses, revealing critical gaps in AI-assisted mental health tools.
Mental disorders are highly prevalent worldwide, but the shortage of psychiatrists and the inherent subjectivity of interview-based diagnosis create substantial barriers to timely and consistent mental-health assessment. Progress in AI-assisted psychiatric diagnosis is constrained by the absence of benchmarks that simultaneously provide realistic patient simulation, clinician-verified diagnostic labels, and support for dynamic multi-turn consultation. We present LingxiDiagBench, a large-scale multi-agent benchmark that evaluates LLMs on both static diagnostic inference and dynamic multi-turn psychiatric consultation in Chinese. At its core is LingxiDiag-16K, a dataset of 16,000 EMR-aligned synthetic consultation dialogues designed to reproduce real clinical demographic and diagnostic distributions across 12 ICD-10 psychiatric categories. Through extensive experiments across state-of-the-art LLMs, we establish key findings: (1) although LLMs achieve high accuracy on binary depression--anxiety classification (up to 92.3%), performance deteriorates substantially for depression--anxiety comorbidity recognition (43.0%) and 12-way differential diagnosis (28.5%); (2) dynamic consultation often underperforms static evaluation, indicating that ineffective information-gathering strategies significantly impair downstream diagnostic reasoning; (3) consultation quality assessed by LLM-as-a-Judge shows only moderate correlation with diagnostic accuracy, suggesting that well-structured questioning alone does not ensure correct diagnostic decisions. We release LingxiDiag-16K and the full evaluation framework to support reproducible research at https://github.com/Lingxi-mental-health/LingxiDiagBench.
