Search papers, labs, and topics across Lattice.
This paper introduces Edge-cloud Speech Recognition and Translation (ESRT), a framework for privacy-preserving and bandwidth-efficient many-to-many speech-to-text translation (S2TT) using MLLMs. ESRT employs an edge-cloud split inference architecture, processing speech on-device with a lightweight encoder and transmitting compressed features to the cloud, reducing bandwidth by 10x and preventing voiceprint leakage. To address English-centric biases, the authors use multi-task weighted curriculum learning with data balancing, achieving state-of-the-art many-to-many S2TT performance on the FLEURS dataset across 45 languages.
Achieve state-of-the-art many-to-many speech translation across 45 languages while slashing bandwidth by 10x and preventing voiceprint leakage.
Multimodal large language models (MLLMs) have demonstrated significant potential for speech-to-text translation (S2TT). However, existing deployment paradigms face critical challenges: pure on-device models suffer from resource constraints, while centralized cloud systems incur severe privacy risks and bandwidth bottlenecks by transmitting raw voice data. Furthermore, most models exhibit English-centric biases, restricting many-to-many translation scaling. In this paper, we propose Edge-cloud Speech Recognition and Translation (ESRT), a privacy-preserving and bandwidth-efficient collaborative edge-cloud MLLM framework. Specifically, we design an edge-cloud split inference architecture that retains a lightweight speech encoder and adapter on the device, transmitting only highly compressed intermediate features to the cloud. This fundamentally prevents voiceprint leakage and reduces bandwidth requirements by up to 10$\times$. To overcome English-centric bottlenecks, we introduce a multi-task weighted curriculum learning strategy with data balancing to ensure robust cross-lingual consistency. Extensive experiments on the FLEURS dataset demonstrate that our models, ESRT-4B and ESRT-12B, achieve state-of-the-art many-to-many S2TT performance across 45 languages ($45 \times 44$ directions). Code and models are released to facilitate reproducible, privacy-aware MLLM S2TT research. The code and models are released at https://github.com/yxduir/esrt.
