Search papers, labs, and topics across Lattice.
The paper introduces ELMoE-3D, a hybrid hardware-software co-design framework leveraging hybrid-bonding (HB) to accelerate Mixture-of-Experts (MoE) model serving on-premise. It introduces Elastic Self-Speculative Decoding (Elastic-SD), which exploits the intrinsic elasticity of MoE models along expert and bit axes to act as both an expert cache and a self-draft model. The framework achieves significant speedups and energy efficiency gains compared to naive MoE serving and prior accelerator baselines by utilizing a 3D-stacked architecture and LSB-augmented bit-sliced execution.
Unlock 6.6x faster and 4.4x more energy-efficient on-premise MoE serving with a hardware-software co-design that turns MoE's memory bottleneck into a strength.
Mixture-of-Experts (MoE) models have become the dominant architecture for large-scale language models, yet on-premises serving remains fundamentally memory-bound as batching turns sparse per-token compute into dense memory activation. Memory-centric architectures (PIM, NMP) improve bandwidth but leave compute underutilized under MoE's low arithmetic intensity at high batch sizes. Speculative decoding (SD) trades idle compute for fewer target invocations, yet verification must load experts even for rejected tokens, severely limiting its benefit in MoE especially at low batch sizes. We propose ELMoE-3D, a hybrid-bonding (HB)-based HW-SW co-designed framework that unifies cache-based acceleration and speculative decoding to offer overall speedup across batch sizes. We identify two intrinsic elasticity axes of MoE-expert and bit-and jointly scale them to construct Elastic Self-Speculative Decoding (Elastic-SD), which serves as both an expert cache and a strongly aligned self-draft model accelerated by high HB bandwidth. Our LSB-augmented bit-sliced architecture exploits inherent redundancy in bit-slice representations to natively support bit-nested execution. On our 3D-stacked hardware, ELMoE-3D achieves an average $6.6\times$ speedup and $4.4\times$ energy efficiency gain over naive MoE serving on xPU across batch sizes 1-16, and delivers $2.2\times$ speedup and $1.4\times$ energy efficiency gain over the best-performing prior accelerator baseline.
