Search papers, labs, and topics across Lattice.
This paper introduces MAESTRO, a novel structured pruning framework specifically designed for sparsely-activated Mixture-of-Experts (MoE) language models, which addresses the inefficiencies of existing pruning methods that fail to account for the interdependencies in MoE routing. By modeling expert activation trajectories as Ergodic Markov chains, MAESTRO provides a globally aware importance heuristic that significantly enhances the pruning process. The framework demonstrates a performance retention improvement of up to 10.61% under a strict 50% compression regime across various domains, while also reducing cross-task variance, thereby improving generalization in diverse applications.
MAESTRO prunes MoE models more effectively by leveraging the interdependencies of expert routing, achieving superior performance retention and consistency across tasks.
Sparsely-activated Mixture-of-Experts (MoE) language models achieve remarkable inference efficiency by activating only a small fraction of parameters per token, yet their full expert banks reside in memory at all times, creating a prohibitive deployment bottleneck. Existing structured pruning methods, largely designed for dense transformers, assess expert importance using locally derived heuristics that are blind to the interdependent nature of MoE routing. We introduce MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting), a structured pruning framework designed for MoE architectures that models autoregressive expert activation trajectories as Ergodic Markov chains whose stationary distributions encode cross-layer dependencies, yielding a globally aware importance heuristic. Evaluated across five diverse domains including Safety, Bias, and Ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention under a strict 50% compression regime, while exhibiting substantially lower cross-task variance, indicating that global, routing-congruent pruning produces models that generalize more consistently across heterogeneous tasks.
NVIDIA