Search papers, labs, and topics across Lattice.
This paper addresses the challenge of multi-talker speech recognition by developing an LLM-based system that efficiently trains on limited real-recorded data while ensuring high accuracy in speaker attribution. The authors introduce a dual-encoder architecture, a feature interleaving format, a length-aware speaker ID loss, and an adaptive threshold strategy for ASR loss computation, which collectively enhance the balance between ASR and diarization tasks. The proposed system demonstrates significant performance improvements, achieving relative gains of 18% on the AliMeeting corpus and 24% on the Aishell4 corpus compared to existing open-source baselines.
Achieving up to 24% improvement in multi-talker speech recognition accuracy with limited training data could redefine efficiency in ASR systems.
Multi-talker speech recognition is often addressed by combining automatic speech recognition (ASR) and speaker diarization in a pipeline system. Recently, LLM-based approaches have shown promise by jointly modeling semantic and speaker information, but they typically require large-scale multi-talker corpora that are costly to annotate. In this paper, we investigate how to efficiently train an LLM-based system with limited real-recorded data while maintaining high accuracy in speaker attribution. We propose several strategies: (1) a dual-encoder architecture to extract semantic and speaker features, (2) a feature interleaving format to merge these features as the inputs to the LLM, (3) a length-aware speaker ID loss to enhance diarization capability, and (4) an adaptive threshold strategy for ASR loss computation to mitigate hallucinations caused by speech overlaps. These strategies balance training between ASR and diarization tasks. Our system outperforms open-source baseline approaches, achieving relative improvements of 18% on the AliMeeting corpus and 24% on the Aishell4 corpus.
