Search papers, labs, and topics across Lattice.
This paper investigates the robustness of instrument grounding in music audio-language models by introducing a novel diagnostic benchmark that extends beyond traditional binary question-answering tasks. The authors reveal that high accuracy in binary QA does not necessarily indicate reliable audio grounding, as models often display biases such as option-position and temporal response errors. These findings advocate for a multi-axis approach to evaluation, highlighting the limitations of current benchmarks in assessing model performance comprehensively.
High binary QA accuracy in music audio-language models can mask significant biases and errors in instrument grounding, revealing the need for more nuanced evaluation methods.
Recent music audio-language models achieve high accuracy on instrument question-answering benchmarks, but it remains unclear whether this reflects robust audio grounding or benchmark-specific shortcuts. In this paper, we introduce an OpenMIC-derived diagnostic benchmark sequence for instrument grounding in music audio-language models, extending binary instrument-presence QA to genre-prior-reduced examples, confusable instrument discrimination, longer audio context, and temporal localization. Across these settings, high binary QA accuracy often fails to predict model behavior: models can exhibit option-position bias, confusable-instrument errors, and temporal response bias. These results suggest that instrument grounding should be evaluated with multi-axis diagnostic benchmarks rather than a single aggregate accuracy.
