Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for evaluating the safety of LLMs in mental health support by pairing AI psychotherapists with simulated patient agents that have dynamic cognitive-affective models. The framework assesses therapy session simulations against a quality of care and risk ontology, identifying iatrogenic risks such as validating patient delusions and failing to de-escalate suicide risk. A large-scale simulation (N=369 sessions) involving six AI agents and 15 patient personas revealed critical safety gaps, and an interactive data visualization dashboard was validated with stakeholders to audit AI psychotherapy.
LLMs in mental health support can inadvertently validate patient delusions ("AI Psychosis") and fail to de-escalate suicide risk, revealing critical safety gaps that demand simulation-based clinical red teaming.
Large Language Models (LLMs) are increasingly utilized for mental health support; however, current safety benchmarks often fail to detect the complex, longitudinal risks inherent in therapeutic dialogue. We introduce an evaluation framework that pairs AI psychotherapists with simulated patient agents equipped with dynamic cognitive-affective models and assesses therapy session simulations against a comprehensive quality of care and risk ontology. We apply this framework to a high-impact test case, Alcohol Use Disorder, evaluating six AI agents (including ChatGPT, Gemini, and Character.AI) against a clinically-validated cohort of 15 patient personas representing diverse clinical phenotypes. Our large-scale simulation (N=369 sessions) reveals critical safety gaps in the use of AI for mental health support. We identify specific iatrogenic risks, including the validation of patient delusions ("AI Psychosis") and failure to de-escalate suicide risk. Finally, we validate an interactive data visualization dashboard with diverse stakeholders, including AI engineers and red teamers, mental health professionals, and policy experts (N=9), demonstrating that this framework effectively enables stakeholders to audit the"black box"of AI psychotherapy. These findings underscore the critical safety risks of AI-provided mental health support and the necessity of simulation-based clinical red teaming before deployment.
