Search papers, labs, and topics across Lattice.
This paper introduces an encoder-only multi-talker automatic speech recognition (MT-ASR) framework that distills semantic priors from large language models (LLMs) into the encoder. The approach uses a post-encoder separator with serialized CTC for talker-ordered transcripts and an LLM-based SOT objective to regularize mixed-speech representations. A Talker-Count Head is introduced to predict the number of speakers and select the appropriate decoding branch, leading to improved performance, especially in three-talker scenarios, with reduced real-time factor (RTF).
Encoder-only multi-talker ASR can now rival LLM-based systems in accuracy while drastically reducing computational cost, thanks to a novel distillation approach and talker-count routing.
Large language models (LLMs) provide strong semantic priors that can improve multi-talker automatic speech recognition (MT-ASR), but using an LLM as an autoregressive decoder is computationally expensive and remains fragile under heavy overlap. In this paper, we propose an encoder-only MT-ASR framework that adapts an LLM to multi-talker conditioning and distills its semantic guidance into the encoder during training, while retaining fast CTC-style decoding at inference. Our model employs a post-encoder separator with serialized CTC to produce talker-ordered transcripts, and leverages an adapted LLM-based SOT objective as a multi-talker-aware teacher signal to explicitly regularize mixed-speech representations. To further support variable numbers of talkers, we introduce a Talker-Count Head that predicts the talker count and dynamically selects the appropriate decoding branch. Experiments on LibriMix show that the proposed encoder-only model achieves comparable performance to LLM-based systems in the two-talker condition, while delivering significant improvements in the three-talker condition with significant small RTF.
