Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of persona-conditioned Large Language Models (LLMs) in simulating diverse demographic perspectives for hate speech annotation. By evaluating inter-group disagreement, in-group sensitivity, and vicarious prediction, the authors find that no single model consistently captures these dimensions, with performance varying significantly across models. Notably, vicarious prompting with Llama 3.1 achieves the highest cross-group agreement, suggesting it may be the most reliable approach for aligning automated annotations with human judgments in this context.
Vicarious prompting with Llama 3.1 offers a breakthrough in achieving cross-group agreement for hate speech annotation, closely mirroring human disagreement patterns.
Hate speech detection is inherently subjective: people from different demographic groups perceive the same content very differently. Collecting enough annotations from multiple demographic groups is costly and difficult to scale. Persona-conditioned Large Language Models (models prompted to adopt a specific demographic identity) have been proposed as a way to simulate diverse perspectives at scale. But do they actually reflect how different groups disagree? We evaluate three aspects of human social judgement: (i) whether personas from different groups disagree in human-like ways (inter-group disagreement), (ii) whether they become more sensitive when content targets their own identity (in-group sensitivity), and (iii) whether they can accurately predict how another group would react (vicarious prediction). Our results show that no model consistently captures all three dimensions, and performance is highly model-dependent and does not emerge reliably from minimal identity prompts alone. However, vicarious prompting with Llama 3.1 yields the highest cross-group agreement in most demographic axes and provides the closest overall approximation to human disagreement patterns, indicating that this configuration may provide a more reliable setting for automatic annotation aligned with human judgements.