Search papers, labs, and topics across Lattice.
This study introduces DelusionEval, a novel evaluation protocol designed to assess large language models (LLMs) for behaviors that may exacerbate user delusions, based on real-world conversation histories from individuals experiencing psychological harm. By analyzing 12,591 messages from 589 unique conversation histories, the research reveals that the propensity for LLMs to engage in delusion-linked behaviors is significantly influenced by the context of prior messages rather than model size or reasoning capabilities. Notably, the rate of failing to discourage self-harm increases from 30.0% to 41.1% when additional context is provided, highlighting the critical role of conversation history in LLM safety evaluations.
Extending context in conversations can significantly amplify the risk of LLMs promoting delusional behaviors, challenging assumptions about model size and reasoning capabilities.
Mental health professionals have raised concerns about risks of psychological harm from interaction with large language models (LLMs), including"delusional spirals"in which concerning human and LLM behaviors reinforce each other over time. With growing public use of LLM-powered chatbots, there is an urgent need to build evaluations grounded in real-world episodes of psychological harm experienced by users. We developed DelusionEval, an evaluation protocol that tests a model's tendencies to exhibit behaviors linked to promoting user delusions. We prompt each model with 589 unique conversation histories from 18 participants, comprising 12,591 messages from users who experienced delusions and psychological harm. We find that the tendency of an evaluated LLM to exhibit delusion-linked behavior does not reliably correlate with model size, release date, or the presence of test-time reasoning. However, extending the context of prior messages substantially increases rates of delusion-linked behaviors, providing evidence for the importance of context in LLM safety evaluation. For example, the rate of failing to discourage self-harm when the user expresses suicidal ideation increases from 30.0% to 41.1% when an additional 350 messages are prepended to the conversation history. All model families (e.g., GPT, Claude) exhibit substantial rates of delusion-linked behaviors. Within families, later, larger, or higher-reasoning models are not uniformly better across all behavior categories. Our results raise concerns regarding the potential psychological impact of LLMs and the need for more rigorous studies of real-world human-AI interaction.
Stanford HAI