Search papers, labs, and topics across Lattice.
This paper introduces MSU-Bench, a benchmark designed to evaluate speaker-centric understanding in multi-speaker conversational scenarios, addressing the limitations of existing benchmarks that focus primarily on single-speaker tasks. The authors developed a two-tier framework encompassing 16 tasks and 2,300 QA instances, utilizing a Gemini-assisted annotation pipeline with human verification to ensure high validity and reliability of the data. Experimental results highlight significant performance gaps across various model families, particularly in complex speaker grounding and reasoning, underscoring the need for improved methodologies in multi-speaker SLU.
Multi-speaker conversational understanding is critically under-evaluated, with MSU-Bench revealing that even leading models struggle with complex speaker grounding tasks.
Spoken Language Understanding (SLU) is moving from task-specific pipelines toward large audio language models (LALMs) that generate natural-language responses. However, existing speech benchmarks mainly focus on single-speaker settings or isolated subtasks, leaving speaker-centric understanding in realistic multi-speaker conversations insufficiently evaluated. We introduce MSU-Bench, a diagnostic benchmark for multi-speaker conversational understanding, covering 16 speaker-centric tasks and 2,300 QA instances in a two-tier framework from speaker grounding to dialogue reasoning. We build a Gemini-assisted annotation and QA generation pipeline with human-in-the-loop verification, achieving high QA validity and strong agreement between human answers and verified labels. We further analyze speaker-referencing schemes and diagnostic error types to reveal bottlenecks in speaker grounding and reasoning. Experiments reveal clear gaps across model families, with closed-source systems leading overall but all models still facing challenges in complex speaker grounding and multi-speaker reasoning. The benchmark annotations, metadata, and evaluation scripts will be available at the GitHub repository: https://github.com/ASLP-lab/MSU-Bench.
