Search papers, labs, and topics across Lattice.
This study investigates the unintended consequences of preprocessing-based stereotype mitigation methods in NLP, revealing that while these techniques can reduce stereotypes for targeted groups, they may inadvertently exacerbate stereotyping for other demographics. The research spans various model architectures and preprocessing strategies, demonstrating that standard evaluation benchmarks often overlook these detrimental shifts. Attention-rollout analysis indicates that these side effects occur without significant changes in attention flow, complicating the understanding of their mechanisms and highlighting the need for more nuanced evaluation practices.
Mitigating stereotypes in NLP can backfire, leading to increased bias against other groups, a phenomenon often missed by standard evaluation metrics.
Preprocessing-based methods for stereotype mitigation, such as pre-/post-training on debiased corpora, are widely used in NLP. While these approaches reduce measurable stereotypes for targeted groups, we find they often induce unintended shifts-side effects, where stereotyping or counter-stereotyping can increase relative to neutral baselines for other demographics, including across unrelated demographic categories. We demonstrate these side effects across two model families (encoder-only and decoder-only), multiple preprocessing strategies (removing stereotypical sentences, removing group mentions, and swapping group references), and both pre- and post-training at different data scales on Wikipedia. Standard benchmarks frequently miss these shifts. Using attention-rollout analysis, we observe that such side effects are not accompanied by large changes in attention flow, complicating mechanistic explanations. We discuss implications for evaluation, provide actionable diagnostics, and argue for side-effect-aware, transparent mitigation practices.
