Search papers, labs, and topics across Lattice.
This paper introduces MentalHospital, a virtual environment designed to evaluate LLM performance in psychiatric clinical encounters by simulating the S.O.A.P. workflow using data from 1,193 de-identified psychiatric EHR cases. The evaluation employs a dual-track protocol that combines objective assessments against EHR references with subjective evaluations of clinical process quality, supported by the MentalEval system, which utilizes domain-specific evaluators for various clinical competencies. Results indicate that while LLMs show promise, they lag significantly behind clinicians in objective psychiatric competence, particularly in mental status assessment, highlighting critical areas for improvement in AI-assisted psychiatric care.
LLMs fall short of clinician performance in psychiatric evaluations, trailing by over 37 percentage points in objective competence.
Large language models (LLMs) have shown strong performance on isolated psychiatric tasks, including dialogue, diagnosis, and treatment planning, yet existing benchmarks rarely simulate complete psychiatric clinical encounters. We introduce $\textbf{MentalHospital}$, a virtual evaluation environment for LLM-based psychiatric clinical encounters. MentalHospital instantiates the Subjective Interviewing, Objective Examination, Diagnostic Assessment, and Treatment Planning (S.O.A.P.) workflow, using skill-augmented standardized patients constructed from 1,193 de-identified psychiatric electronic health record (EHR) cases spanning all major ICD-11 categories and 76 disorders. Each encounter is assessed through a dual-track protocol that combines objective comparison against EHR-derived references with subjective assessment of clinical process quality. To scale specialist judgment, we develop $\textbf{MentalEval}$, five domain-specific evaluators covering communication empathy, interviewing professionalism, clinical-note quality, diagnostic rigor, and treatment appropriateness, trained with rubric-grounded SFT and expert-guided DPO. Survey responses from 22 clinicians support MentalHospital's clinical fidelity (3.88/5), while MentalEval achieves strong expert alignment with an average QWK of 0.944. Benchmarking shows that even the strongest LLM trails clinicians by 37.28 percentage points in objective psychiatric competence, with mental status assessment as a key bottleneck.
