Search papers, labs, and topics across Lattice.
This study introduces CompanionHarm, a benchmark dataset designed to evaluate harms in multi-turn conversations between users and AI companions, specifically Replika, comprising 2,111 conversations and 14,051 utterances. The dataset features annotations across 13 categories of harmful behavior, revealing that while multi-turn context enhances harm detection, current large language models (LLMs) struggle with contextual integration and harm severity calibration. Notably, the research highlights significant annotator disagreement influenced by political affiliation and conversation dynamics, underscoring the complexity of interpreting socio-emotional harms in AI interactions.
Multi-turn context significantly boosts harm detection in AI conversations, yet LLMs still falter in understanding nuanced relational dynamics.
As AI companions become increasingly embedded in everyday life, there is an urgent need to detect harms that emerge in social and emotional human-AI interactions. Yet research in this area is constrained by the lack of real-world, multi-turn conversational datasets for operationalizing and evaluating harms that are relational and contextual. In this work, we introduce CompanionHarm, a publicly available benchmark dataset comprising 2,111 real-world, multi-turn conversations (14,051 utterances) between users and the AI companion Replika. 7,016 AI utterances were annotated independently by three annotators across 13 harmful behavior categories grounded in a taxonomy of AI companion harms, and the dataset includes both aggregated labels and annotator-level labels to support model evaluation and systematic disagreement analysis. Evaluations of seven large language models (LLMs) show that harm detection using multi-turn conversational context outperforms detection based on isolated utterances, although current LLMs still struggle to consistently integrate contextual cues, calibrate harm severity, and interpret relational boundaries. We also find substantial annotator disagreement for context-dependent harmful behaviors, with disagreement varying according to annotators'political affiliation, conversation length, and the utterance's position. Together, CompanionHarm provides a foundation for detecting socio-emotional harms in multi-turn human-AI conversations and for rigorously examining how such harms are interpreted by both humans and LLMs. Our dataset is available at https://github.com/HanMeng2004/CompanionHarm.
