Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the conditions under which incorporating annotator demographic information improves hate speech detection. Through analysis of disagreement metrics, training size, and demographic coverage, they identify regimes where demographic features provide gains, specifically those with low training disagreement, high test disagreement, and sufficient training data. They then introduce a gated demographic residual model that selectively incorporates demographic information, showing improved performance on high disagreement examples.
Demographic information isn't a magic bullet for hate speech detection; its usefulness hinges on specific data characteristics and a model that knows when to listen.
Demographic information is often used to model annotator perspectives in subjective tasks such as hate speech detection, but its benefit is inconsistent: it improves performance in some settings and behaves as noise in others. This paper asks when demographic features help. We analyze demographic gain as a function of both data split properties and modeling frameworks. For data splits, we measure annotator disagreement, namely how often annotators assign different labels to the same example, along with training size and train-test demographic coverage. We find that demographic gains concentrate in regimes with low training disagreement, high test disagreement, fine-grained ambiguity measurement, sufficient training data, and greater demographic overlap. Motivated by these regimes, we introduce a gated demographic residual model that treats demographics as a selective adjustment to text-only predictions. Experiments on MHS and POPQUORN show that this design is effective, especially on high disagreement or low confidence examples. Overall, our results suggest that demographics should not be assumed useful by default; their value depends jointly on the data regime and the modeling framework.