Search papers, labs, and topics across Lattice.
This paper introduces AudioSpan, a comprehensive benchmark designed to evaluate large audio-language models (LALMs) on audio comprehension tasks spanning durations from 10 minutes to over 2 hours, accompanied by 3,240 questions across cognitive levels of perception, understanding, and reasoning. The benchmark employs two distinct question generation methods鈥擭ative QA and Anchor QA鈥攁llowing for a nuanced assessment of model performance in handling long-form audio content. Results reveal that LALMs struggle primarily with the perception phase, particularly in extracting relevant information from lengthy audio, highlighting a critical gap in current model capabilities as audio length increases.
LALMs face significant challenges in audio comprehension, especially in extracting relevant facts from lengthy signals, with performance deteriorating as audio duration increases.
General audio comprehension now covers speech, sound, and music over durations from seconds to hours, driven by large audio-language models (LALMs) that are increasingly omni-modal. Yet the benchmarks that test them still rely on clips of seconds, where scores saturate and models converge; recent long-form efforts extend duration but evaluate long audio much as short clips are. We introduce AudioSpan, a benchmark that spans both duration and depth: it pairs audio from 10 minutes to over 2 hours with 3,240 questions across three cognitive levels, namely perception, understanding, and reasoning. Two paths supply the questions, differing in how question content is sourced and how ground truth is obtained. Native QA extracts questions from the audio's content, posing each as a multiple-choice item and an open-ended one graded by detailed rubrics. Anchor QA instead injects ground truth, planting acoustic anchors into the audio and building a perception-to-reasoning chain scored only to the first error. A fully automated pipeline constructs every item through structured captioning, QA generation, and adversarial critic feedback. Evaluating 12 LALMs on AudioSpan, we find the hard part comes before reasoning: distilling a few relevant facts from a long, redundant signal. This difficulty grows with audio length and falls hardest on perception, especially temporal grounding. AudioSpan is available at https://huggingface.co/datasets/holvan/AudioSpan.
