Search papers, labs, and topics across Lattice.
The paper introduces FilBBQ, a new bias benchmark extending the BBQ dataset to the Philippine context, designed to evaluate sexist and homophobic biases in question-answering language models. It employs a four-phase development process including culturally aware translation and new template construction to generate over 10,000 prompts. The study demonstrates the presence of biases related to emotion, domesticity, queer interests, and polygamy in Filipino language models, while also highlighting the variability of bias scores across different random seeds.
Language models trained in Filipino exhibit significant sexist and homophobic biases specific to the Philippine context, even when accounting for response instability across multiple seeds.
With natural language generation becoming a popular use case for language models, the Bias Benchmark for Question-Answering (BBQ) has grown to be an important benchmark format for evaluating stereotypical associations exhibited by generative models. We expand the linguistic scope of BBQ and construct FilBBQ through a four-phase development process consisting of template categorization, culturally aware translation, new template construction, and prompt generation. These processes resulted in a bias test composed of more than 10,000 prompts which assess whether models demonstrate sexist and homophobic prejudices relevant to the Philippine context. We then apply FilBBQ on models trained in Filipino but do so with a robust evaluation protocol that improves upon the reliability and accuracy of previous BBQ implementations. Specifically, we account for models' response instability by obtaining prompt responses across multiple seeds and averaging the bias scores calculated from these distinctly seeded runs. Our results confirm both the variability of bias scores across different seeds and the presence of sexist and homophobic biases relating to emotion, domesticity, stereotyped queer interests, and polygamy. FilBBQ is available via GitHub.
