Search papers, labs, and topics across Lattice.
This paper introduces Router-weighted Expert Activation Merging (REAM), a novel approach to compress Mixture-of-Experts (MoE) LLMs by merging experts instead of pruning them. REAM groups and merges expert weights based on router weights, aiming to better preserve performance compared to pruning methods like REAP. Experiments across multiple MoE LLMs on question answering and generation tasks demonstrate that REAM can outperform baselines and achieve performance comparable to uncompressed models, revealing a trade-off between multiple-choice and generative performance based on the calibration data mix.
Merging experts in MoE LLMs can actually *improve* performance compared to pruning, offering a new path to compression that preserves capabilities.
Mixture-of-Experts (MoE) large language models (LLMs) are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.
Mila