Search papers, labs, and topics across Lattice.
The paper introduces HumMusQA, a new benchmark dataset for evaluating music understanding in Large Audio-Language Models (LALMs), consisting of 320 expert-validated, human-written questions designed to probe complex audio comprehension. Benchmarking six state-of-the-art LALMs on HumMusQA reveals limitations in their ability to truly perceive and interpret music, highlighting the need for more rigorous evaluation methodologies. The dataset also allows for testing robustness to uni-modal shortcuts, providing insights into the models' reliance on non-musical cues.
LALMs still struggle to truly "hear" music, as revealed by a new expert-curated benchmark that exposes their reliance on non-musical shortcuts.
The evaluation of music understanding in Large Audio-Language Models (LALMs) requires a rigorously defined benchmark that truly tests whether models can perceive and interpret music, a standard that current data methodologies frequently fail to meet. This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training, arguing that such focused, manual curation is superior for probing complex audio comprehension. To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts.
