Search papers, labs, and topics across Lattice.
This paper adapts Granite-speech, a speech-aware LLM, for speaker-attributed ASR (SAA) by introducing speaker cluster identification tags trained jointly with SAA. The key innovation is the use of these speaker cluster tags (e.g., "[Speaker 1 cluster 42]:") to improve SAA accuracy. They also propose a data augmentation method using artificially concatenated multi-speaker conversations to overcome training data limitations, achieving state-of-the-art results on multiple benchmarks compared to traditional diarization-then-ASR pipelines.
Speaker-attributed ASR gets a serious boost from jointly training speaker cluster tags within a speech-aware LLM, outperforming traditional pipelines.
Speaker-Attributed Automatic Speech Recognition (SAA) enhances traditional ASR systems by incorporating relative speaker identity tags directly into the transcript (e.g., [Speaker 1]:, [Speaker 2]:). In this work, we extend the capabilities of Granite-speech, a state-of-the-art speech-aware Large Language Model (LLM) originally trained for transcription and translation. We demonstrate that it can be effectively adapted for SAA with only minimal architectural changes. Our core contribution is the introduction of speaker cluster identification tags (e.g., [Speaker 1 cluster 42]:) which are jointly trained with SAA to significantly improve accuracy. To address limitations in training data, we propose a data augmentation method that uses artificially concatenated multi-speaker conversations. Our approach is evaluated across multiple benchmarks and shows superior performance compared to conventional pipelines that sequentially perform speaker diarization followed by ASR.
