Search papers, labs, and topics across Lattice.
This paper investigates LLM alignment with human hate speech judgments by decomposing the task into ten subjective attributes (e.g., dehumanization, violence, sentiment) and evaluating Llama 3.1 and Qwen 2.5 models. The analysis reveals a consistent divergence: LLMs align well with behavioral attributes but invert evaluative ones. By combining attribute-level predictions via confidence-weighted Ridge regression, the authors achieve a significantly improved $R^2$ of 0.71 in reconstructing continuous hate speech scores compared to direct prompting.
LLMs consistently invert human judgments on evaluative attributes of hate speech (e.g., respect, sentiment), even while aligning on behavioral aspects, suggesting a fundamental misalignment in subjective understanding.
Hate speech annotation is costly, subjective, and prone to annotator disagreement, making large-scale dataset construction challenging. We systematically analyze how well large language models (LLMs) align with human judgments across ten theoretically grounded subjective attributes, such as dehumanization, violence, and sentiment, evaluating both small and large variants of Llama 3.1 and Qwen 2.5. Our analysis reveals a consistent split across all models: behaviorally explicit dimensions (insult, humiliate, attack-defend) correlate strongly with human annotations, while evaluative dimensions (respect, sentiment, hate speech) are systematically inverted. Demographic persona conditioning reduces model confidence without improving alignment. Building on these insights, we propose combining attribute-level LLM predictions via a confidence-weighted Ridge regression to reconstruct continuous hate speech scores from the Measuring Hate Speech corpus, achieving $R^2$ of up to 0.71 and outperforming direct prompting baselines, demonstrating that structured attribute decomposition recovers a richer and more human-aligned signal than end-to-end label prediction alone.
