Search papers, labs, and topics across Lattice.
This paper introduces DiaScriber, an end-to-end model for multi-speaker automatic speech recognition (MSASR) that effectively integrates diarization and transcription to address the challenges of speaker identification and content transcription in complex audio environments. By leveraging a pretrained speech large language model (Qwen3.5-Omni) and employing a three-stage training strategy, DiaScriber significantly improves performance in scenarios characterized by rapid turn-taking and overlapping speech. The results indicate that DiaScriber not only outperforms existing methods but also exhibits remarkable generalization capabilities in previously unseen multi-speaker contexts.
DiaScriber achieves unprecedented accuracy in multi-speaker scenarios, overcoming the challenges of overlapping speech and rapid transitions.
Multi-speaker automatic speech recognition (MSASR) aims to jointly predict content transcriptions, speaker identities, and timestamps, thereby addressing the key question of"who spoke what and when"and holds substantial practical value in real-world multi-speaker scenarios. However, MSASR still encounters considerable challenges in the presence of fast turn transitions, overlapping speech, and complex, diverse multi-speaker scenarios. In this work, we propose DiaScriber, an end-to-end multi-speaker diarization and transcription model built on a speech large language model. We first construct diverse data pipelines to cover a wide variety of multi-speaker scenarios and their complexities, including validation and refinement, turn-transition and overlapping-speech simulation, and multimodal annotation. Furthermore, DiaScriber is developed based on the pretrained version of Qwen3.5-Omni through a three-stage training strategy involving continual pretraining, supervised fine-tuning, and reinforcement learning. Experiments show that DiaScriber achieves superior performance over comparison methods across extensive multi-speaker scenario test sets and demonstrates outstanding generalization ability in unseen multi-speaker scenarios.
