Search papers, labs, and topics across Lattice.
This paper introduces DraftExpert, a novel framework for self-speculative decoding in large Mixture-of-Experts (MoE) language models, specifically designed for latency-sensitive end-device applications. By training lightweight draft experts that leverage residual, logit/token, and router-agreement signals from a frozen target MoE, the approach effectively balances accuracy and efficiency during inference. The results demonstrate a 1.45x improvement in decode throughput and significantly higher draft acceptance rates, making it a compelling solution for optimizing MoE inference in constrained environments.
DraftExpert achieves a 1.45x boost in decoding throughput while maintaining high draft acceptance rates, revolutionizing MoE inference for end-device applications.
Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU. In this setting, self-speculative decoding faces a new bottleneck: increasing the draft expert set improves accuracy but triggers extra expert loading, while cheap small-footprint drafts have low acceptance; moreover, verifying a multi-token block activates the union of target experts and is no longer close to one target step. We propose DraftExpert, an expansion-aware self-speculative decoding framework for expert-offloaded MoE inference. DraftExpert trains one lightweight accelerator-resident draft expert per layer by self-distilling residual, logit/token, and router-agreement signals from the frozen target MoE. At inference time, it uses a fixed-footprint shared+top-1+draft-expert drafter together with confidence--expansion truncation and target-expert prefetching, while final tokens are still exactly verified by the target model. On DeepSeek-V2-Lite and Moonlight-16B-A3B across CPU-GPU and Flash-NPU offload, DraftExpert improves decode throughput by 1.45x on average, raises draft acceptance to 84~87%, and achieves 86~88% prefetch hit rates.
