Search papers, labs, and topics across Lattice.
This paper introduces Cachemir, a novel framework for fully homomorphic encrypted (FHE) inference of generative LLMs that addresses the inefficiency of integrating KV caches in existing FHE solutions. Cachemir achieves this by developing HE packing algorithms tailored for KV cache utilization, an interleaved replicated packing algorithm for efficient vector-matrix multiplications, and an augmented bootstrapping placement strategy to minimize bootstrapping costs. Experiments show that Cachemir significantly outperforms state-of-the-art FHE inference frameworks like MOAI and THOR, achieving up to 67x speedup and generating tokens for Llama-3-8B in under 100 seconds on GPU.
Achieve practical FHE inference for Llama-3-8B with sub-100 second token generation by cleverly integrating KV caching, leaving prior art in the dust.
Generative large language models (LLMs) have revolutionized multiple domains. Modern LLMs predominantly rely on an autoregressive decoding strategy, which generates output tokens sequentially and employs a key-value cache (KV cache) to avoid redundant computation. However, the widespread deployment of LLMs has raised serious privacy concerns, as users are feeding all types of data into the model, motivating the development of secure inference frameworks based on fully homomorphic encryption (FHE). A major limitation of existing FHE-based frameworks is their inability to effectively integrate the KV cache, resulting in prohibitively high latency for autoregressive decoding. In this paper, we propose Cachemir, a KV Cache Accelerated Homomorphic Encrypted LLM Inference Regime to overcome this limitation. Cachemir comprises three key technical contributions: 1) a set of novel HE packing algorithms specifically designed to leverage the computational advantages of the KV cache; 2) an interleaved replicated packing algorithm to efficiently compute the vector-matrix multiplications that result from using the KV cache in Transformer linear layers; and 3) an augmented bootstrapping placement strategy that accounts for the KV cache to minimize bootstrapping cost. We demonstrate that Cachemir achieves $48.83\times$ and $67.16\times$ speedup over MOAI (ICML'25) and THOR (CCS'25) respectively on CPU and consumes less than 100 seconds on GPU to generate an output token for Llama-3-8B.
