Search papers, labs, and topics across Lattice.
This paper introduces the MSRT framework, which employs a resource-aware Mixture of Speech Encoders (MoSE) to tackle the curse of multilinguality in speech-to-text translation (S2TT). By utilizing an explicit language router to allocate utterances to specialized expert encoders, the approach effectively enhances performance across high-, medium-, and low-resource languages, achieving state-of-the-art results with a 4B-parameter model. The implementation of a five-stage curriculum learning strategy significantly reduces the data requirements, demonstrating that only 10 hours of paired S2TT data per language is necessary for effective alignment, particularly benefiting low-resource languages.
Breaking the curse of multilinguality, this framework boosts low-resource speech performance while preserving high-resource capabilities, all with minimal data requirements.
Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT). However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from the curse of multilinguality: languages at different resource levels compete for limited representation capacity, leading to strong high-resource performance but substantial degradation on low-resource speech. To address this problem and improve multilingual consistency, we propose MSRT, a novel framework built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses an explicit language router to assign each utterance to an appropriate expert encoder. A frozen expert preserves high-resource language capabilities, while a trainable expert adapts to and specializes in medium- and low-resource languages. We further introduce a five-stage curriculum learning strategy that substantially reduces data dependence, requiring only 10 hours of paired S2TT data per language for effective alignment. We conduct extensive experiments on 45 languages, systematically evaluating all $45 \times 44$ translation directions. Our 4B-parameter model achieves state-of-the-art performance, outperforming substantially larger baselines. Empirical analyses show that MoSE improves high-, medium-, and low-resource languages simultaneously, with the largest gains on low-resource speech, thereby breaking the curse of multilinguality without compromising high-resource performance. To support future multilingual S2TT research, we release our code and models.
