Search papers, labs, and topics across Lattice.
This paper introduces the Music I Care About Meta-Benchmark (MusICA-MetaBench), a novel framework for automated, on-demand benchmarking of Multimodal Large Language Models (MLLMs) in music perception across various modalities. By utilizing structured symbolic representations and predefined question templates, the framework generates multiple-choice questions that assess music perception skills aligned with music pedagogy. The authors validate their approach using the ChoraleBricks dataset, demonstrating that their benchmarks effectively measure music perception competencies and facilitate reliable model comparisons across different musical formats.
Automated benchmarks can now evaluate MLLMs' music perception skills across diverse modalities, ensuring more reliable assessments than ever before.
Music represents a cornerstone of human culture, existing digitally across diverse modalities, including audio, symbolic encodings (e.g., MIDI, MusicXML), and sheet music. Despite the advancement of Multimodal Large Language Models (MLLMs), current music benchmarks face three major limitations. First, large static benchmarks are resource-intensive to evaluate, and it remains unclear how their results transfer to diverse kinds of music beyond those included in the benchmark. Second, benchmarks claiming to measure"music understanding"often fail to require music perception. Third, they do not support systematic performance comparisons across musical modalities. To overcome these issues, we introduce the Music I Care About Meta-Benchmark (MusICA-MetaBench), a framework that automatically derives on-demand benchmarks directly from user-provided data. By leveraging structured symbolic representations (e.g., MusicXML) and our pre-defined question templates, we build multiple-choice question-answer pairs that probe music perception competencies, aligned with music pedagogy, across audio, music notation images, and symbolic files. We demonstrate our framework with the ChoraleBricks dataset, and experimentally determine benchmark sizes that ensure statistically reliable model comparisons for this setup. By comparing against text-only and white-noise baselines, we show our questions do measure music perception. Ultimately, MusICA-MetaBench represents a significant advancement in the cross-modal assessment of music perception for MLLMs. By proposing a dataset-specific benchmarking paradigm, it enables efficient on-demand evaluation of music perception capabilities.
