Search papers, labs, and topics across Lattice.
This paper presents a novel human-LLM collaborative annotation framework to construct EspanStereo, a Spanish-language stereotype dataset that encompasses stereotypes from various Spanish-speaking countries. By leveraging LLMs to generate candidate stereotypes and validating them with in-culture annotators, the study effectively identifies nuanced, region-specific biases that are often overlooked in English-centric datasets. The evaluation of Spanish-supporting LLMs using EspanStereo reveals significant variations in stereotypical behavior across different countries, underscoring the necessity for culturally grounded assessments in AI systems.
Significant variations in stereotypical behavior across Spanish-speaking countries reveal the limitations of English-centric stereotype datasets in AI.
Research on stereotypes in large language models (LLMs) has largely focused on English-speaking contexts, due to the lack of datasets in other languages and the high cost of manual annotation in underrepresented cultures. To address this gap, we introduce a cost-efficient human-LLM collaborative annotation framework and apply it to construct EspanStereo, a Spanish-language stereotype dataset spanning multiple Spanish-speaking countries across Europe and Latin America. EspanStereo captures both well-documented stereotypes from prior literature and culturally specific biases absent from English-centric resources. Using LLMs to generate candidate stereotypes and in-culture annotators to validate them, we demonstrate the framework's effectiveness in identifying nuanced, region-specific biases. Our evaluation of Spanish-supporting LLMs using EspanStereo reveals significant variation in stereotypical behavior across countries, highlighting the need for more culturally grounded assessments. Beyond Spanish, our framework is adaptable to other languages and regions, offering a scalable path toward multilingual stereotype benchmarks. This work broadens the scope of stereotype analysis in LLMs and lays the groundwork for comprehensive cross-cultural bias evaluation.
