Search papers, labs, and topics across Lattice.
This paper introduces Cost-Aware Expert Execution (CAEE), a framework designed to optimize inference performance in Mixture-of-Experts (MoE) architectures by addressing both token-level expert importance and system-level execution costs. By employing calibrated cost models, CAEE selectively prunes low-importance experts and redistributes their contributions without incurring additional data movement costs. The results demonstrate an 8%-18% reduction in end-to-end inference latency across various deployment settings while keeping accuracy loss below 1%.
CAEE slashes inference latency in MoE models by intelligently pruning low-value experts, achieving efficiency gains without sacrificing accuracy.
Mixture-of-Experts (MoE) architectures enable language models to achieve unprecedented scale via sparse activation. However, their inference performance is often limited by data movement bottlenecks. Two coupled challenges exacerbate this limtation: (1) Importance-Agnostic Cost: Low-contribution experts incur nearly uniform memory and transfer costs, resulting in a low cost-to-benefit ratio and wasting critical bandwidth; (2) System-Level Imbalance: Multi-device deployments are universally bottlenecked by the slowest device, meaning that local reductions on one device may yield no improvement in end-to-end latency. We propose Cost-Aware Expert Execution (CAEE), a hardware-guided runtime framework that jointly optimizes for token-level expert importance and system-level execution cost. CAEE uses lightweight, calibrated cost models to estimate hardware overhead, selectively prunes low-importance, high-cost experts, and redistributes their contributions via a low-overhead compensation mechanism, avoiding extra data movement. Evaluations on the 671B DeepSeek-R1 model show that CAEE can reduce end-to-end inference latency by 8\%-18\% across diverse deployment settings, including expert offloading and on-device execution on multi-device systems, while maintaining a model accuracy drop of less than 1\%.
