Search papers, labs, and topics across Lattice.
The paper introduces VibeVoice-ASR-Streaming, a novel LLM-based end-to-end model that integrates speaker attribution and automatic speech recognition (ASR) for real-time applications. By interleaving fixed-size audio chunks with lookahead audio and previous text, the model enables immediate identification of speakers as speech is processed, eliminating the need for a separate diarization stage. The 7B model demonstrates superior performance, achieving the lowest average word and character error rates across five evaluation sets and excelling in speaker attribution across most evaluation settings.
Real-time speaker-attributed ASR is now feasible with VibeVoice-ASR-Streaming, achieving unmatched accuracy while processing speech on-the-fly.
Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce''who said what''as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.
