Search papers, labs, and topics across Lattice.
This paper introduces distributed prompt caching, a method for accelerating local LLM inference on resource-constrained edge devices by sharing intermediate processing states across multiple devices. The approach leverages partial matching to exploit prompt similarity and employs a Bloom-filter-based catalog to minimize communication overhead associated with state sharing. Experiments using the Gemma-3 270M model on Raspberry Pi Zero 2W with the MMLU dataset demonstrate a 93.12% reduction in TTFT and a 50.07% reduction in TTLT.
Edge devices can achieve a 93% reduction in time-to-first-token for local LLM inference by cooperatively caching and sharing intermediate processing states.
Since local LLM inference on resource-constrained edge devices imposes a severe performance bottleneck, this paper proposes distributed prompt caching to enhance inference performance by cooperatively sharing intermediate processing states across multiple low-end edge devices. To fully utilize prompt similarity, our distributed caching mechanism also supports partial matching. As this approach introduces communication overhead associated with state sharing over a wireless network, we introduce a Bloom-filter-based data structure, referred to as a catalog, to determine whether a remote server possesses the desired internal states, thereby suppressing unnecessary communication. Experiments using the Gemma-3 270M model and the MMLU dataset on the Raspberry Pi Zero 2W platform demonstrate that the proposed approach reduces TTFT (Time to First Token) and TTLT (Time to Last Token) by 93.12% and 50.07% on average, respectively.
