Search papers, labs, and topics across Lattice.
This paper investigates the performance discrepancies between matrix extensions, like Arm Scalable Matrix Extension (SME), and conventional CPU cores during LLM inference, revealing that different operations exhibit varying arithmetic intensities and memory bandwidth competition. By employing a roofline-based characterization, the authors develop SMEPilot, an inference engine that intelligently selects between CPU-only, SME-only, or hybrid execution strategies tailored to specific operator shapes. The results show that SMEPilot achieves up to 3.94脳 improvement in end-to-end inference performance across multiple LLMs and platforms, highlighting the potential of optimized execution strategies in leveraging modern CPU architectures.
SMEPilot achieves up to 3.94脳 faster LLM inference by intelligently balancing workloads between CPU and SME units based on operator characteristics.
Modern CPUs increasingly integrate matrix extensions, such as Arm Scalable Matrix Extension (SME), that provide high-throughput matrix execution within the CPU. For LLM inference, however, these units are not a universal replacement for conventional CPU cores: prefill, decode, attention, and KV-cache operations expose different arithmetic intensities, vector behavior, and layout requirements, while SME units and CPU cores still compete for shared memory bandwidth. This paper studies this mismatch through a roofline-based characterization of SME-enabled CPUs and uses the resulting model to guide operator-level execution choices. We present SMEPilot, an LLM inference engine that selects CPU-only, SME-only, or cooperative SME+CPU execution for each operator shape. SMEPilot partitions matrix work across SME and CPU cores at tile granularity, overlaps SME-suitable matrix stages with CPU-suitable vector stages in attention, and maintains layout state so packed tensor representations are reused rather than repeatedly rebuilt on critical paths. Across Llama-3.2-3B, Qwen3-4B, and Qwen3-30BA3B on phone, PC, and server platforms, SMEPilot improves end-to-end inference performance by up to 3.94$\times$.
