Search papers, labs, and topics across Lattice.
This paper introduces mzCache, an innovative on-device memory management system for Large Language Models (LLMs) designed to optimize inference in dynamic multitasking environments on mobile devices. By implementing a restoration-oriented memory management strategy that partitions LLM memory into fine-grained shared buffers, mzCache enables concurrent eviction and restoration, significantly enhancing responsiveness during application switches. The system achieves a remarkable 2.1-5.5脳 reduction in Time-to-First-Token compared to traditional storage-backed methods, demonstrating its practical effectiveness in real-world multitasking scenarios.
mzCache slashes Time-to-First-Token by up to 5.5脳 in multitasking environments, revolutionizing on-device LLM responsiveness.
On-device mobile Large Language Model (LLM) inference is gaining significant attention. However, mobile devices operate in highly dynamic multitasking environments where users frequently switch between applications. This creates memory pressure, forcing LLM memory (model weights and KV cache) to be evicted by the operating system. When a new inference request arrives, the inference system must restore the evicted memory through slow storage reads or recompute the entire KV cache, severely degrading responsiveness. To address this, we present mzCache, an on-device LLM inference system with specialized memory management for multitasking environments. Under unpredictable memory pressure, mzCache elastically evicts LLM memory and leverages the unified memory of mobile SoCs to enable zero-wait inference on the GPU with concurrent CPU-side restoration. mzCache realizes this through restoration-oriented memory management: LLM memory is partitioned into fine-grained shared buffers to enable partial eviction and restoration with concurrent cross-processor access, while hybrid swap and backward-out eviction policies ensure low-latency restoration from any eviction state. Implemented on llama.cpp and deployed as an Android application, mzCache achieves 2.1-5.5$\times$ reduction in Time-to-First-Token compared to storage-backed partial offload and demonstrates its effectiveness in real multitasking scenarios.
BAIR