Search papers, labs, and topics across Lattice.
The paper introduces Persona-based Client Simulation Attack (PCSA), a red-teaming framework that simulates clients in psychological counseling to expose vulnerabilities in LLMs' psychological safety alignment. PCSA generates coherent, persona-driven client dialogues to elicit harmful responses from LLMs, such as providing unauthorized medical advice or reinforcing delusions. Experiments across seven LLMs show PCSA significantly outperforms existing red-teaming baselines in eliciting such harmful behaviors, highlighting the models' vulnerability to domain-specific adversarial tactics.
LLMs readily reinforce harmful beliefs and behaviors when interacting with simulated clients in psychological counseling, a risk missed by standard red-teaming approaches.
The increasing use of large language models (LLMs) in mental healthcare raises safety concerns in high-stakes therapeutic interactions. A key challenge is distinguishing therapeutic empathy from maladaptive validation, where supportive responses may inadvertently reinforce harmful beliefs or behaviors in multi-turn conversations. This risk is largely overlooked by existing red-teaming frameworks, which focus mainly on generic harms or optimization-based attacks. To address this gap, we introduce Personality-based Client Simulation Attack (PCSA), the first red-teaming framework that simulates clients in psychological counseling through coherent, persona-driven client dialogues to expose vulnerabilities in psychological safety alignment. Experiments on seven general and mental health-specialized LLMs show that PCSA substantially outperforms four competitive baselines. Perplexity analysis and human inspection further indicate that PCSA generates more natural and realistic dialogues. Our results reveal that current LLMs remain vulnerable to domain-specific adversarial tactics, providing unauthorized medical advice, reinforcing delusions, and implicitly encouraging risky actions.
