Search papers, labs, and topics across Lattice.
This paper introduces VQKV, a training-free KV cache compression method for LLMs that uses vector quantization to represent floating-point values with integer indices, achieving high compression ratios and reconstruction fidelity. VQKV attains an 82.8% compression ratio on LLaMA3.1-8B while maintaining 98.6% of baseline performance on LongBench. This enables 4.3x longer generation lengths within the same memory footprint.
Squeeze your LLM's KV cache by 82% without significant performance loss using VQKV's novel vector quantization approach.
The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches for KV cache compression typically rely on low-rank approximation or scalar quantization, which fail to simultaneously achieve high compression ratios and high reconstruction fidelity. We propose VQKV, a novel, training-free method introducing vector quantization (VQ) to obtain highly compressed KV representations while preserving high model fidelity, allowing for the representation of thousands of floating-point values with just a few integer indices. As a result, VQKV achieves an 82.8\% compression ratio on LLaMA3.1-8B while retaining 98.6\% of the baseline performance on LongBench and enabling 4.3x longer generation length on the same memory footprint.
