Search papers, labs, and topics across Lattice.
This paper introduces K-VEC, a novel coverage-aware key-value (KV) cache eviction strategy designed to enhance the efficiency of large language models (LLMs) during inference. By addressing the performance drop associated with existing eviction methods, K-VEC prioritizes token coverage through a cross-head and cross-layer coverage module, significantly improving predictive accuracy. Evaluations on 16 LongBench subsets reveal that K-VEC achieves up to a 10.35 point improvement over traditional methods while maintaining the same eviction rate and memory constraints.
Prioritizing token coverage in KV-cache eviction can boost LLM performance by over 10 points without increasing memory usage.
Large language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs. However, deploying LLMs is costly, not only due to the high computational demands of quadratic complexity of self-attention and auto-regressive generation, but also because of the significant memory overhead required for storing the key-value (KV) cache during inference. To reduce the memory cost, existing KV-cache eviction strategies leverage the sparsity in attention to selectively store a subset of tokens. While reducing the memory footprint, such approaches show a considerable drop in performance, especially in tasks that require long-context reasoning. We identify that the drop in performance is linked to a reduction in the coverage of unique tokens. Additionally, we theoretically show that reduced coverage limits the mutual information between inputs and outputs, thereby impairing predictive accuracy. To this end, we introduce K-VEC, a novel coverage-aware KV-cache eviction strategy that prioritizes token coverage while evicting tokens in the cache. K-VEC introduces a cross-head and a cross-layer coverage module to enhance token retention across attention heads and model layers, mitigating performance degradation caused by low coverage. Evaluated on 16 LongBench subsets, K-VEC exhibit up to 10.35 points improvement over the existing methods under the same eviction rate and memory constraint. Comprehensive evaluations validate the effectiveness of our approach and demonstrate its potential for efficient LLM deployment in resource-constrained settings.
