Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces ESCUCHA, the first benchmark specifically designed for evaluating Spanish speech understanding in large audio language models (LALMs) across diverse acoustic conditions. This benchmark includes 1,000 human-curated questions paired with 162.9 hours of real-world audio, emphasizing reasoning and linguistic diversity through various Spanish accents and non-normative speech. Results from benchmarking state-of-the-art models indicate significant performance gaps compared to human understanding, highlighting the need for improved LALM capabilities in realistic settings.
Spanish speech understanding lags behind, with state-of-the-art models showing substantial performance gaps compared to human listeners in diverse acoustic conditions.
As large audio language models (LALMs) advance, robust evaluation frameworks have become essential. In this context, Spanish speech understanding under realistic acoustic conditions has received particularly little attention. We introduce ESCUCHA, the first Spanish speech understanding benchmark designed to evaluate LALMs across heterogeneous acoustic conditions and reasoning abilities. ESCUCHA comprises 1,000 human-curated questions paired with audio, totaling 162.9 hours sourced directly ``from the wild''rather than drawn from existing datasets, with durations ranging from a few seconds to over 80 minutes. The benchmark emphasizes reasoning, spanning 9 perceptual and 10 reasoning categories, and it captures linguistic diversity through multiple Spanish accents and non-normative speech. ESCUCHA further includes multi-audio questions, spoken questions, and audio instructions, and it flags which questions support open-ended evaluation. Benchmarking several state-of-the-art multimodal and speech models reveals substantial performance gaps relative to trained humans.