Search papers, labs, and topics across Lattice.
This paper introduces SDARE-Bench, a novel benchmark designed to evaluate large language models (LLMs) on their ability to detect and respond to stigma in both dyadic and group dialogues. The study reveals that LLMs struggle significantly with identifying stigma components, particularly in group settings, where stigma expression rates can reach as high as 97.5%. These findings highlight a critical safety vulnerability in LLMs, emphasizing the need for improved models that can handle socially complex conversational contexts more effectively.
LLMs exhibit alarming levels of stigma expression in group dialogues, with detection rates plummeting and unrealistic advice proliferating under social pressure.
Large Language Models (LLMs) are increasingly used in advice seeking and decision making that may affect social judgements. Despite stigma's profound effects on people and communities, benchmarks remain scarce. Existing general-domain evaluations typically rely on static prompts and fixed-format tasks, overlooking conversational contexts and audience effects in everyday communication. To address these gaps, we introduce SDARE-Bench, the first scenario-based benchmark evaluating both stigma detection and open-ended response generation in LLMs, comprising 1,138 dyadic queries and 1,388 group dialogue. Empirical results across 8 LLMs consistently demonstrate poor identification of stigma components, especially in group dialogues. In open-ended response generation, stigma expression was substantially higher in group settings than in dyadic, with weaker resistance to stigma and more unrealistic advice. Responses were evaluated using a classifier trained on 1,392 human annotated responses. In constructed group pressure settings, stigma expression rates further increased to a striking average of 97.5%. Our findings identify stigma response as a recurring LLM safety vulnerability, especially in socially complex conversational contexts.
