Search papers, labs, and topics across Lattice.
This paper introduces EXAM$^2$, a comprehensive benchmark designed to evaluate multilingual and multimodal audio understanding across six languages and various audio modalities, including speech, sound, and music. By integrating visual information with diverse audio inputs, EXAM$^2$ facilitates a more realistic assessment of scene-aware audio reasoning and cross-modal comprehension. The evaluation of state-of-the-art large audio language models (LALMs) reveals significant performance gaps, while the proposed Gemma3n-EXAM$^2$ model demonstrates up to 21.7% improvement in multimodal tasks, underscoring the benchmark's potential to advance research in this area.
Multilingual and multimodal audio understanding is critically under-evaluated, with EXAM$^2$ revealing up to 21.7% performance gaps in current models.
Recent large audio language models (LALMs) have achieved impressive progress in audio understanding. However, existing evaluations remain largely constrained to English and narrow audio domains. Prior benchmarks typically focus on a single audio modality, i.e., speech, sound, or music, limiting the systematic investigation into how these models generalize across diverse visual scenarios. In this paper, we introduce EXAM$^2$, a benchmark for multilingual and multimodal audio understanding spanning six languages and multiple modalities, including speech, sound, music, mixed-audio settings, and visual images. By incorporating visual information alongside heterogeneous audio inputs, EXAM$^2$ enables more realistic evaluation of scene-aware audio reasoning and cross-modal comprehension. EXAM$^2$ comprises $5,667$ multiple-choice questions, $22,614$ image instances, and $135,684$ multilingual translations. We evaluate state-of-the-art open-source and proprietary LALMs as well as multimodal LLMs, revealing substantial performance gaps in multilingual and cross-modal understanding. Furthermore, we propose Gemma3n-EXAM$^2$, a lightweight fusion-model fine-tuned on EXAM$^2$-train, achieves up to $12.4\%$ improvement in multilingual settings and $21.7\%$ gains in multimodal evaluation over a strong baseline. Empirical results establish EXAM$^2$ as a challenging benchmark and pioneer future multilingual and multimodal audio intelligence research.
