Search papers, labs, and topics across Lattice.
This paper introduces a cache-resident execution model for large language model (LLM) inference that optimizes the use of GB-scale last-level caches in modern server CPUs. By decoupling weight-centric operators from attention and key-value (KV) cache management, the model minimizes synchronization overhead and enhances cache residency, leading to significant performance improvements. The prototype implementation demonstrates a remarkable speedup of up to 13.9x in time-per-output-token across various model configurations, showcasing the potential of this approach for efficient LLM inference on commodity hardware.
Achieving up to 13.9x speedup in LLM inference by leveraging cache-resident execution could redefine performance benchmarks for large models on standard CPUs.
Large language model (LLM) inference is increasingly dominated by data movement across the memory hierarchy. Recent 3D-stacked cache technologies have enabled GB-scale last-level caches in modern server CPUs, making it possible to keep reusable model weights on chip and exploit cache bandwidth and latency. Achieving this regime is not straightforward: deeper pipelining for weight residency increases in-flight requests and KV-cache footprint, while cache-resident operators make operator-boundary synchronization a visible bottleneck. We present a cache-resident execution model for inference on hierarchical-memory clustered systems. The model separates weight-centric operators from attention and KV-cache management into dedicated resource domains, keeping reusable weights cache-resident while scaling KV capacity independently of pipeline depth. It also relaxes synchronization from operator boundaries to true sub-operator dependencies, reducing coordination overhead in the cache-resident regime. We instantiate this model on a multi-socket CPU cluster with a weight-attention decoupled architecture, locality-aware placement, and a specialized static runtime. The prototype substantially outperforms equally provisioned llama.cpp. On deployed Llama-3.2-3B and Llama-2-7B configurations, it achieves 2.04x-11.51x speedup on time-per-output-token (TPOT). Under a validated analytical model, it further reaches up to 13.9x TPOT speedup across model sizes, context lengths, and batch sizes. These results show that commodity CPUs with GB-scale last-level caches can support efficient LLM inference when execution is organized around cache residency, decoupled state management, and dependency-aware coordination.
