Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces a method for aggressively pruning experts from Mixture-of-Experts LLMs to create smaller, translation-focused models. They leverage expert specialization and multilingual separability to identify and prune irrelevant experts without retraining, achieving significant parameter reduction. With short supervised fine-tuning, they prune up to 75% of experts while maintaining baseline translation performance, and nearly 90% in some cases with acceptable quality.
Translation powerhouses are hiding inside LLMs: prune away 90% of the experts and still get reasonable quality.
Modern large language models (LLMs) achieve state-of-the-art machine translation performance, but they do so as broad generalists largely trained for many tasks and capabilities unrelated to translation. Thus, they are heavily overparameterized for this task, resulting in excessive memory and compute requirements. In this paper, we present a method for aggressively pruning experts from modern mixture-of-experts LLMs while incurring negligible degradation in translation quality. Our approach exploits expert specialization and the separability of multilingual capabilities in LLMs to identify experts irrelevant to translation. And because of the modular nature of MoEs, these can be easily pruned without any training. Without retraining, we are able to prune half of all experts with negligible degradation and 70% with only minor losses. With a very short SFT, we prune 75% of experts while recovering baseline performance, and in some settings remove nearly 90% while maintaining reasonable translation quality. Overall, our results show that translation requires only a fraction of the LLM, enabling substantial compression of the MoE blocks that contain over 90% of parameters.