Search papers, labs, and topics across Lattice.
This study addresses the under-explored issue of anti-queer bias in Dutch language models by creating a culturally and linguistically adapted dataset derived from the English WinoQueer benchmark. Through an online survey with 43 Dutch queer participants, the authors validated existing stereotypes and uncovered new biases, resulting in a dataset of 42,906 sentences. The evaluation revealed that while the average bias score was neutral, significant disparities were found, with some models exhibiting a strong preference for stereotypical sentences, particularly for transgender identities, emphasizing the need for culturally relevant datasets in bias assessment.
Dutch language models exhibit alarming biases, with some favoring stereotypical representations of transgender identities up to 97% of the time.
While English language models have been widely examined for anti-queer bias, Dutch models remain understudied. To address this gap, we developed a culturally and linguistically adapted Dutch dataset based on the English WinoQueer benchmark, containing pairs of stereotypical and counter-stereotypical sentences. To validate and expand it, we conducted an online survey with 43 Dutch queer participants, confirming 145 of 171 stereotypes as culturally relevant and identifying 22 new biases through free-text responses. The final released dataset, comprising 42,906 sentences, was evaluated using a range of Dutch-specific and multilingual models, including both masked language models (MLMs) and autoregressive language models (ARLMs), with bias measured via a score comparing log-likelihoods of stereotypical versus counter-stereotypical sentences. While the mean bias score across models appeared neutral (~50%), closer analysis revealed significant disparities: some models favored stereotypical sentences up to 97% of the time for transgender identities, but only 6% of the time for gay-related pairs, with transgender and non-binary identities consistently receiving the highest bias scores. Our findings highlight the importance of culturally grounded datasets for evaluating and mitigating biases that disproportionately impact marginalized groups in Dutch language models.
