Search papers, labs, and topics across Lattice.
This paper introduces EdgeXpert, a software-hardware co-designed accelerator that enhances memory-efficient inference of Large Language Models (LLMs) on edge devices by effectively combining speculative decoding and mixture-of-experts (MoE) techniques. The proposed method addresses the incompatibility between these techniques by reformulating expert routing to focus on prompt-level expert reuse and employing depth-aware expert coalescing, significantly reducing external memory access (EMA) during feed-forward network (FFN) layers. As a result, EdgeXpert achieves up to 56.3% latency reduction and 44.1% energy savings while maintaining near-baseline accuracy, making it a compelling solution for personalized edge applications.
EdgeXpert slashes LLM inference latency by over 56% while cutting energy use by nearly 45%, all without sacrificing accuracy.
On-device deployment of Large Language Models (LLMs) has become essential for personalized edge applications. A primary bottleneck is external memory access (EMA) in feed-forward network (FFN) layers. Speculative decoding and mixture-of-experts (MoE) are promising solutions. Speculative decoding reduces the number of decoding stages by generating multiple tokens per stage, and MoE minimizes per-stage cost through sparse expert activation. However, there is an incompatibility when combining these two techniques. We propose EdgeXpert, a software-hardware co-designed LLM accelerator that resolves this incompatibility. In the prefill stage, the prompt-wise expert reuse reformulates routing as prompt-level expert reuse rather than independent per-token expert selection. It identifies important tokens using a lightweight encoder, constructs a shared expert set from them, and routes less important tokens with a reduced expert budget to lower expert EMA. In the decode stage, depth-aware expert coalescing exploits the contextual similarity and mutual exclusivity of same-depth candidate tokens. Rather than loading the union of all required channels, EdgeXpert loads only salient channels and applies computational calibration to recover accuracy without additional memory access. Synthesized in Samsung 28nm technology at 800 MHz, EdgeXpert achieves up to 56.3% latency reduction and 44.1% energy reduction compared to prior works, while maintaining near-baseline accuracy.
