Search papers, labs, and topics across Lattice.
This study investigates gender bias in large language models (LLMs) by examining their associations with musical instruments through a newly created multimodal dataset, Symphony-Bias, which encompasses text, vision, and audio. The analysis of ten multimodal models reveals that 92% of the associations align with established social-science findings, indicating a strong reinforcement of gender stereotypes, particularly with instruments like the harp and drums. Notably, the research highlights that the strength of these associations varies by modality, being weakest in audio and strongest in text, which underscores the complex interplay between representation and bias in LLMs.
92% of musical instrument associations in LLMs reinforce existing gender stereotypes, with striking modality-specific variations in bias strength.
Large language models (LLMs) are increasingly embedded in everyday life and widely used for information seeking, raising concerns about their potential to perpetuate social biases and reinforce stereotypes. In this study, we investigate gender bias in LLMs through the lens of their associations with musical instruments. Building on social-science research on the cultural gender-typing of instruments, we introduce Symphony-Bias, a parallel multimodal dataset spanning text, vision, and audio. We evaluate ten multimodal models with diverse architectures and scales across 22 musical instruments, analyzing how they associate each instrument with three gender categories: {male, female, non-binary}, across three modalities: {text, vision, audio}. Our results show that 92\% of instrument-level outcomes align with prior social-science findings, with the harp and drums showing particularly consistent gendered associations across all evaluated models and modalities. We further find that alignment with social stereotypes is weakest in audio, stronger in vision, and strongest in text, suggesting that modality-specific representations can differentially amplify gendered associations with musical instruments.\footnote{The Symphony-Bias dataset will be publicly released upon acceptance of the paper.}
