Search papers, labs, and topics across Lattice.
This paper conducts a comprehensive cross-layer measurement study of mobile Large Language Model (LLM) inference across five frameworks and three hardware backends, revealing critical efficiency bottlenecks. The authors introduce PowerBench, a fine-grained profiling tool that provides backend-specific energy attribution, highlighting that NPUs can experience performance gaps up to 10x due to framework-induced inefficiencies. Key findings include the identification of a phase split where NPUs excel in compute-bound tasks but lag in memory-bound decoding, alongside a proposed best-practice configuration that could reduce energy consumption by up to 54.8% on NPUs.
NPUs can waste up to 40% of energy due to suboptimal configurations, but a new profiling tool reveals how to cut this waste significantly.
Deploying Large Language Models (LLMs) on mobile devices enhances privacy and reduces latency, but is severely bottlenecked by hardware inefficiency. We present the first comprehensive, cross-layer measurement study of mobile LLM inference, uniquely spanning five mainstream frameworks (e.g., llama.cpp, GENIE) and three hardware backends (CPU, GPU, NPU). To enable this analysis, we develop PowerBench, a fine-grained profiling tool that provides the first backend-specific energy attribution, moving beyond traditional device-level measurements. Our study yields three critical insights: (1) Framework-induced performance gaps are substantially amplified on NPUs, reaching up to 10x using custom operators due to divergent offloading and quantization strategies. (2) We identify a distinct phase split where NPUs excel at compute-bound prefilling, while CPUs outperform all other backends in memory-bound decoding. This is driven by the NPU's preference for large, fixed-shape workloads, which conflicts with the small-kernel, dynamic nature of decoding. (3) Backend-specific profiling uncovers substantial scheduling headroom missed by prior work. Suboptimal thread configurations, uncoordinated NPU sleep latencies, and CPU polling intervals result in up to 40% energy waste. Leveraging these findings, we present an energy-oriented best-practice configuration for mobile LLM inference. We estimate that this configuration could reduce energy consumption by up to 54.8% on the NPU backend across three datasets.
Tsinghua AI