Search papers, labs, and topics across Lattice.
Tsinghua AIBeijing National Research Center for InformationSCUXingyun Integrated Circuits CoJun 9, 2026arXiv:2606.10493This paper addresses the performance gaps in local deployment of large Mixture-of-Experts (MoE) models compared to cloud environments, identifying critical issues such as low throughput and poor concurrency. The authors introduce a CPU-GPU hybrid system that significantly enhances prefill throughput and concurrency, achieving cloud-level service level objectives (SLOs) on consumer hardware. Key innovations include stream-loading techniques, expert parallelism, and optimized kernels, resulting in a system capable of processing 45K prompts in 30 seconds with minimal latency increase.
Local MoE inference can now match cloud-level performance, achieving 1,800 tokens/s on consumer GPUs while maintaining original model precision.
Local deployment of large Mixture-of-Experts (MoE) models falls short of the service quality achieved in cloud-scale environments, even under low-concurrency workloads. We identify four key gaps in local MoE inference: reliance on capacity-reduced models (quantized, distilled, rerouted), inability to meet 30-second TTFT for long prefills (more than 12K), sub-baseline decode throughput (under 20 tokens/s), and poor concurrency under mixed prefill-decode and batched decode workloads. We present a CPU-GPU hybrid system that achieves cloud-level SLOs on dual-socket commodity CPUs and consumer GPUs by (1) stream-loading prefill (SLP), boosting prefill throughput to 1,200 tokens/s and enabling 32K prompts within 30 seconds; (2) distributed SLP (DSLP) with SmallEP expert parallelism, reaching 1,800 tokens/s and 45K prompts in 30 seconds on two RTX 5090s; (3) intra-node prefill-decode disaggregation with zero-copy shared weights and a dual-batch attention-MoE overlap scheme, sustaining concurrency with under 15 percent latency increase and 50 percent throughput gains; (4) an AVX-512-optimized FP8 GEMV kernel, enabling native CPU FP8 inference while delivering 4-5x lower CPU latency; and (5) fine-grained CPU parallelism that attains 28 tokens/s on INT4 DeepSeek-V3 and 21.5 tokens/s on intact FP8 V3. Evaluations show our system delivers cloud-level QoS for flagship MoE models on consumer CPU-GPU platforms, reshaping local deployment with intact, original-precision inference and enabling high-quality, cost-effective access without datacenter infrastructure.
