Search papers, labs, and topics across Lattice.
This study addresses the inefficiencies in multilingual neural machine translation (MNMT) caused by large vocabularies, proposing a novel framework that integrates vocabulary pruning with targeted fine-tuning. By reducing the vocabulary size from over 128,000 to approximately 10,000 tokens, the approach achieves a remarkable 60% reduction in memory usage while maintaining competitive translation performance. The pruned M2M100 model not only matches the BLEU score of a dedicated bilingual model but also surpasses it in semantic adequacy and fluency, as indicated by the COMET metric.
Pruning the vocabulary of multilingual models can lead to a 60% memory savings without sacrificing translation quality, challenging the need for large vocabularies in MNMT.
The adoption of large pre-trained multilingual models for neural machine translation (MNMT) faces a major challenge: excessive memory and computational consumption due to overly large vocabularies and embedding layers. Although existing compression methods like pruning, quantization and knowledge distillation reduce parameter redundancy, they mainly preserve the structure of the original vocabulary, thereby leaving a major source of inefficiency unresolved. We propose in this paper a general optimization framework that combines a vocabulary pruning method with a targeted fine-tuning protocol for MNMT models. We evaluate the proposed framework using three models (M2M100, NLLB-200, mBART-50) on the English-Arabic language pair. Our approach reduces the vocabulary size from over 128,000 to approximately 10,000 tokens, enabling a 60% memory saving without any loss in performance. Results show that optimized multilingual models can match or exceed the performance of dedicated bilingual baselines. In particular, the pruned and fine-tuned M2M100 model achieves a competitive BLEU score of 42.04 (against 44.59 for the OPUS-MTen- ar bilingual model) while it significantly outperforms it on the COMET metric (0.8730 vs 0.7911) revealing superior semantic adequacy and fluency.
