Search papers, labs, and topics across Lattice.
This paper introduces CompanionHarm, a benchmark dataset consisting of 2,111 real-world, multi-turn conversations between users and the AI companion Replika, aimed at detecting socio-emotional harms in human-AI interactions. The dataset includes 7,016 AI utterances annotated across 13 categories of harmful behavior, revealing that while multi-turn context improves harm detection in large language models (LLMs), these models still face challenges in integrating contextual cues and calibrating harm severity. Additionally, the study highlights significant annotator disagreement influenced by factors such as political affiliation and conversation dynamics, underscoring the complexity of interpreting harms in AI interactions.
Multi-turn context significantly enhances harm detection in AI conversations, yet LLMs still falter in understanding relational nuances and severity.
As AI companions become increasingly embedded in everyday life, there is an urgent need to detect harms that emerge in social and emotional human-AI interactions. Yet research in this area is constrained by the lack of real-world, multi-turn conversational datasets for operationalizing and evaluating harms that are relational and contextual. In this work, we introduce CompanionHarm, a publicly available benchmark dataset comprising 2,111 real-world, multi-turn conversations (14,051 utterances) between users and the AI companion Replika. 7,016 AI utterances were annotated independently by three annotators across 13 harmful behavior categories grounded in a taxonomy of AI companion harms, and the dataset includes both aggregated labels and annotator-level labels to support model evaluation and systematic disagreement analysis. Evaluations of seven large language models (LLMs) show that harm detection using multi-turn conversational context outperforms detection based on isolated utterances, although current LLMs still struggle to consistently integrate contextual cues, calibrate harm severity, and interpret relational boundaries. We also find substantial annotator disagreement for context-dependent harmful behaviors, with disagreement varying according to annotators'political affiliation, conversation length, and the utterance's position. Together, CompanionHarm provides a foundation for detecting socio-emotional harms in multi-turn human-AI conversations and for rigorously examining how such harms are interpreted by both humans and LLMs. Our dataset is available at https://github.com/HanMeng2004/CompanionHarm.
