Search papers, labs, and topics across Lattice.
This paper introduces AcceptMoE, a verifier-side expert selector that optimizes the efficiency of speculative decoding in mixture-of-experts (MoE) models by dynamically adjusting the number of eligible experts based on cache residency and commitment probabilities. The approach addresses the challenge of excessive expert activation during verification, which can lead to increased computational costs without proportional gains in accuracy. Results show that AcceptMoE achieves a throughput improvement of up to 2.06 times compared to the baseline while maintaining competitive accuracy, significantly reducing host-to-device traffic by up to 77.1%.
AcceptMoE slashes host-to-device traffic by over 73% while boosting throughput by more than double, all without a major accuracy trade-off.
Speculative decoding verifies a tree of draft tokens in one target-model forward pass. For a mixture-of-experts (MoE) target, however, parallel verification can activate the union of the experts selected by all tree nodes, even though only a small subset of those nodes reaches the accepted output. Token count, activated-expert union size, and expert-weight traffic are therefore distinct cost measures: reducing the token workload need not shrink the expert union proportionally, and under offloading, transfer traffic also depends on cache residency. We introduce AcceptMoE, a verifier-side expert selector that combines target-router scores with offline-estimated commitment probabilities and automatically adjusts the number of eligible experts for each verification block, eliminating the need for a user-specified expert budget. Under offloading, AcceptMoE conditions expert eligibility on cache residency instead of predicting natural routes and prefetching the corresponding expert weights. Although constraining target-expert eligibility changes the model distribution, across 12 model-task pairs spanning three MoE targets and four benchmarks, AcceptMoE's mean accuracy is 0.27 percentage points lower than that of EAGLE-3 speculative decoding with natural routing. Served with SGLang at batch size one, it reaches 1.290 times the throughput of this baseline with all expert weights in GPU memory, and 2.06 times under physical expert offloading, while reducing host-to-device traffic by 73.6 percent to 77.1 percent.
