Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces VIBE, a novel framework for evaluating generative biases in Large Audio-Language Models (LALMs) through open-ended tasks using real-world speech. By moving beyond traditional benchmarks that rely on synthetic speech and Multiple-Choice Questions, VIBE captures biases in a more organic manner, revealing significant distributional shifts influenced by gender and accent cues. The evaluation of 12 state-of-the-art LALMs demonstrates that bias magnitude varies significantly depending on the task, highlighting the need for more nuanced assessments of fairness in audio-language applications.
Systematic biases in LALMs can be triggered by subtle cues like gender and accent, revealing a complex landscape of fairness that traditional benchmarks miss.
Large Audio-Language Models (LALMs) are increasingly integrated into daily applications, yet their generative biases remain underexplored. Existing speech fairness benchmarks rely on synthetic speech and Multiple-Choice Questions (MCQs), both offering a fragmented view of fairness. We propose VIBE, a framework that evaluates generative bias through open-ended tasks such as personalized recommendations, using human-recorded speech. Unlike MCQs, our method allows stereotypical associations to manifest organically without predefined options, making it easily extensible to new tasks. Evaluating 12 state-of-the-art LALMs reveals systematic biases in realistic scenarios. Both gender and accent cues trigger statistically significant distributional shifts, and bias magnitude is strongly task-dependent.
NVIDIA