Search papers, labs, and topics across Lattice.
The paper introduces PolySpeech-100, a comprehensive benchmark for evaluating speech understanding across over 100 languages and dialects, addressing critical gaps in existing methodologies that favor high-resource languages and low-level transcription tasks. Utilizing a hybrid construction pipeline that combines human recordings with synthetic speech, the benchmark enables a thorough assessment of state-of-the-art models, revealing that open-source E2E models outperform traditional Cascade systems in handling dialects while also highlighting a significant performance drop for these models on low-resource languages. Additionally, the study uncovers that Chain-of-Thought prompting can negatively impact speech understanding in zero-shot scenarios, indicating a misalignment in current model architectures regarding modality processing.
Open-source E2E models outperform Cascade systems in understanding heavy dialects, but struggle significantly with low-resource languages, revealing critical gaps in current speech model architectures.
While End-to-End (E2E) Speech-Large Language Models (Speech-LLMs) are rapidly evolving, their evaluation methodologies remain limited to the era of simple transcription. Existing benchmarks suffer from three critical limitations: a pronounced bias towards high-resource languages, a focus on low-level recognition (ASR) rather than semantic reasoning, and a neglect of regional dialects. To bridge this gap, we introduce PolySpeech-100, a massive-scale benchmark designed to assess `native-level' speech comprehension across 110 linguistic variants. We employ a novel hybrid construction pipeline that augments gold-standard human recordings with instruction-driven synthetic speech, allowing us to cover 19 distinct Chinese dialects and over 80 low-resource languages. Extensive evaluation of 22 state-of-the-art models (including Gemini-3, GPT-Audio, and Qwen2.5-Omni) yields pivotal insights. First, we demonstrate that open-source E2E models outperform Cascade (ASR+LLM) systems on heavy dialects, proving that direct audio processing preserves critical paralinguistic cues and prosodic features (e.g., intonation, stress) that are often lost in standard transcription. Second, we reveal a significant performance gap: while commercial models maintain robustness, open-source models suffer catastrophic degradation on low-resource languages. Finally, counter-intuitively, we observe that under standard zero-shot settings, Chain-of-Thought prompting frequently degrades speech understanding performance for most evaluated models, revealing a potential modality alignment gap in current architectures. PolySpeech-100 establishes a rigorous standard for the next generation of inclusive, omni-capable Speech-LLMs. The data, demo, and code are publicly available at https://github.com/YoungSeng/PolySpeech-100.
Tsinghua AI