Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper evaluates anti-LGBTQ biases in multilingual language models, focusing on a newly created benchmark dataset that captures cultural nuances in German and English. By assessing eight language models of varying sizes and architectures, the study reveals that these models often perpetuate anti-queer stereotypes, with significant differences in bias manifestation across identities and model types. Fine-tuning on community-sourced and progressive media content shows promise in reducing bias, though results are inconsistent, underscoring the need for culturally adaptive approaches in bias evaluation.
Language models not only replicate anti-queer stereotypes but also show inconsistent responses to bias mitigation strategies across different identities and architectures.
While gender and racial biases in language models have been widely studied, anti-LGBTQ biases remain underexplored, particularly beyond English. Existing benchmarks often do not capture cultural and linguistic variation and rely on gender representations. This paper introduces a multilingual German-English benchmark dataset for the evaluation of anti-LGBTQ biases in language models. It combines community-sourced stereotypes from German-speaking queer individuals with a German translation of WinoQueer. The data is used to evaluate eight language models across sizes and architectures and explore mitigation through fine-tuning on community and progressive media content. Results show that language models reproduce anti-queer stereotypes, with variation across identities and models. Differences between the translated and community-based data highlight the importance of cultural adaptation for multilingual bias evaluation. Fine-tuning reduces bias on average, but not consistently across models and identities. Warning: This text contains examples of anti-queer hateful language and stereotypes.