Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the memory bottleneck of LLM KV caches by proposing MultiAlloc, a profiling-guided framework for budget-aware precision allocation. MultiAlloc adaptively distributes limited quantization resources between layer-wise precision enhancement and token-wise preservation, informed by offline profiling of layer sensitivity and token importance. Experiments show MultiAlloc consistently outperforms existing methods across diverse models and benchmarks under matched memory budgets.
LLMs can achieve superior KV cache compression by intelligently allocating quantization budgets between layer-wise and token-wise strategies, outperforming uniform or single-granularity approaches.
The Key-Value (KV) cache constitutes a dominant source of memory consumption during large language model (LLM) inference, particularly in long-context and complex reasoning scenarios. Existing KV cache quantization methods have explored different compression granularities, including layer-wise precision adjustment and token-level information preservation. In practical deployment, however, the key challenge is not merely compression, but how to allocate limited quantization resources among heterogeneous precision preservation strategies under a fixed KV cache memory budget. We propose MultiAlloc, a profiling-guided precision allocation framework for KV cache quantization that adaptively distributes limited quantization resources between layer-wise precision enhancement and token-wise preservation. MultiAlloc employs lightweight offline profiling to characterize layer sensitivity and token importance, enabling adaptive precision allocation under a fixed memory constraint. Guided by the profiled memory--accuracy trade-off, it automatically derives an effective precision allocation strategy without online allocation search, and applies the resulting policy during inference. Extensive experiments on reasoning and long-context benchmarks demonstrate that MultiAlloc consistently improves over layer-wise, token-wise, and recent multi-granularity KV cache quantization methods under matched memory budgets across diverse models, including Mistral, Llama, DeepSeek-Qwen, and QwQ.