Search papers, labs, and topics across Lattice.
This paper introduces MetaNet, a support-set controller that optimizes expert allocation in Mixture-of-Experts (MoE) models by predicting layer-specific expert-retention thresholds and routing biases based on task conditions. The approach allows for dynamic adjustment of expert activation, resulting in significant reductions in the number of active experts while maintaining competitive accuracy levels on benchmark tasks. Specifically, MetaNet achieves a 40% reduction in active experts with comparable accuracy and demonstrates transferability to other evaluation tasks without retraining.
Activating 62% fewer experts without sacrificing significant accuracy could revolutionize the efficiency of Mixture-of-Experts models in real-world applications.
Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token computation sparse. In many deployed MoEs, the number of active experts is fixed across layers and tasks, although layer roles and expert redundancy vary with depth and demand varies with difficulty. Existing approaches address only part of this setting: layer-wise allocations are usually determined offline and reused for all tasks, while token-level methods vary expert activation using local routing signals without task-level context. We propose MetaNet, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias. The backbone, experts, and router remain frozen. On DeepSeek-MoE-16B-Chat, MetaNet provides a tunable accuracy-expert-activation trade-off. Relative to fixed k=6, a conservative setting activates 3.61 experts on average (40% fewer) and achieves comparable MMLU accuracy (0.489 vs. 0.474), whereas an aggressive setting activates 2.28 experts on average (62% fewer) with accuracy approximately 3.7 percentage points lower. The MMLU-trained controller also transfers to C-Eval without retraining, activating 2.90 experts on average (52% fewer than fixed k=6) at 0.386 accuracy.
