Search papers, labs, and topics across Lattice.
PSI-Bench is introduced as an automatic evaluation framework for depression patient simulators, providing clinically grounded diagnostics across turn, dialogue, and population levels. Benchmarking seven LLMs across two simulator frameworks reveals that current simulators produce overly long, lexically diverse responses, exhibit reduced variability, resolve emotions too quickly, and follow a uniform negative-to-positive trajectory. A human study validates the benchmark's alignment with expert clinical judgment, highlighting the framework's utility in guiding future simulator development.
Current depression patient simulators are more like Pollyannas than patients, resolving negative emotions too quickly and following a predictable trajectory from negative to positive.
Patient simulators are gaining traction in mental health training by providing scalable exposure to complex and sensitive patient interactions. Simulating depressed patients is particularly challenging, as safety constraints and high patient variability complicate simulations and underscore the need for simulators that capture diverse and realistic patient behaviors. However, existing evaluations heavily rely on LLM-judges with poorly specified prompts and do not assess behavioral diversity. We introduce PSI-Bench, an automatic evaluation framework that provides interpretable, clinically grounded diagnostics of depression patient simulator behavior across turn-, dialogue-, and population-level dimensions. Using PSI-Bench, we benchmark seven LLMs across two simulator frameworks and find that simulators produce overly long, lexically diverse responses, show reduced variability, resolve emotions too quickly, and follow a uniform negative-to-positive trajectory. We also show that the simulation framework has a larger impact on fidelity than the model scale. Results from a human study demonstrate that our benchmark is strongly aligned with expert judgments. Our work reveals key limitations of current depression patient simulators and provides an interpretable, extensible benchmark to guide future simulator design and evaluation.
