Search papers, labs, and topics across Lattice.
This paper introduces H-SAGE, a novel framework for Multi-Talker Automatic Speech Recognition (MTASR) that addresses the limitations of existing Mixture-of-Experts (MoE) approaches by incorporating a Speaker-Aware Global Encoder and an Overlap-Aware Loss. By capturing long-term dependencies and providing explicit acoustic guidance, H-SAGE improves expert collaboration in high-overlap speech scenarios. Experimental results on LibriSpeechMix show significant performance gains over strong baselines, highlighting the effectiveness of the proposed methods in complex transcription tasks.
Explicit acoustic guidance in H-SAGE leads to significant performance improvements in multi-talker ASR, especially in challenging overlapping speech conditions.
Multi-talker Automatic Speech Recognition (MTASR) faces significant challenges in accurately transcribing overlapping speech, particularly under complex high-overlap conditions. While recent Mixture-of-Experts (MoE) approaches have shown promise, they typically rely on frame-independent routing that leads to temporal myopia, and depend solely on the downstream ASR objective, which results in implicit and ungrounded representation learning. To address these limitations, we propose Holistic Speaker-Aware Guided Experts (H-SAGE) for MoE-based MTASR. Specifically, we introduce a Speaker-Aware Global Encoder to capture long-term dependencies, supervised by an auxiliary Overlap-Aware Loss that explicitly guides the model to discern acoustic states. Furthermore, we design a Holistic Gating Mechanism to arbitrate expert selection by jointly evaluating global context and local details. Experiments on LibriSpeechMix demonstrate that H-SAGE achieves consistent improvements over strong baselines, particularly in complex scenarios, validating that explicit acoustic guidance effectively enhances expert collaboration. Our code can be found at https://github.com/NKU-HLT/H-SAGE.
