Search papers, labs, and topics across Lattice.
The paper introduces AUDITA, a new large-scale audio question answering dataset designed to evaluate robust auditory reasoning beyond superficial acoustic recognition. AUDITA features human-authored trivia questions grounded in real-world audio, incorporating challenging distractors and long-range temporal dependencies to prevent models from relying on shortcut strategies. Experiments show that state-of-the-art audio QA models achieve less than 9% accuracy on AUDITA, significantly underperforming humans (32%), highlighting their limitations in genuine audio reasoning.
SOTA audio QA models are getting punked by trivia questions a toddler could answer, revealing a stark gap between current capabilities and true audio understanding.
Existing audio question answering benchmarks largely emphasize sound event classification or caption-grounded queries, often enabling models to succeed through shortcut strategies, short-duration cues, lexical priors, dataset-specific biases, or even bypassing audio via metadata and captions rather than genuine reasoning Thus, we present AUDITA (Audio Understanding from Diverse Internet Trivia Authors), a large-scale, real-world benchmark to rigorously evaluate audio reasoning beyond surface-level acoustic recognition. AUDITA comprises carefully curated, human-authored trivia questions grounded in real-world audio, designed to stress robust auditory reasoning through challenging distractors and long-range temporal dependencies, using probing queries that cannot be answered from isolated text or sound cues alone. Human average accuracy of 32.13% shows both the challenge of the task while demonstrating meaningful comprehension of the audio. In stark contrast, state of-the-art audio question answering models perform poorly, with average accuracy below 8.86%. Beyond raw accuracy, we apply Item Response Theory (IRT) to estimate latent proficiency, question difficulty, and expose systematic deficiencies of the models and data.
