Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This research introduces a projector-based framework that enhances multilingual automatic speech recognition (ASR) by addressing challenges in multilingual generalization and modality alignment. By integrating a Mixture of Experts (MoE) architecture with a Continuous Integrate-and-Fire (CIF) mechanism for dynamic downsampling, the proposed method significantly improves performance over existing baseline models. The findings indicate that this approach could lead to more accurate and robust LLM-based ASR systems capable of better handling diverse languages and modalities.
Leveraging a Mixture of Experts and dynamic downsampling, this framework achieves unprecedented multilingual ASR performance improvements that could redefine language model applications in speech recognition.
The rapid progress of large language models (LLMs) has opened up a new frontier for automatic speech recognition (ASR), making their effective integration a critical and challenging research direction. To this end, this work proposes a projector-based LLM-ASR framework targeting the key challenges of multilingual generalization and modality alignment. Our approach incorporates a Mixture of Experts (MoE) architecture to improve cross-lingual adaptability, and a Continuous Integrate-and-Fire (CIF) mechanism for dynamic downsampling and modality alignment. Experimental results show that the combination of these components yields substantial performance improvements, surpassing strong baseline models. The proposed method represents a step toward building more accurate, robust, and generalizable LLM-based ASR systems.
Tsinghua AI