Search papers, labs, and topics across Lattice.
This paper presents MosaicKV, a dynamic two-dimensional key-value (KV) cache compression system designed to enhance the efficiency of long-context language model (LLM) serving. By intelligently compressing both the sequence and channel dimensions of the KV cache while minimizing accuracy loss, MosaicKV achieves significant memory savings and performance improvements. The evaluation demonstrates that MosaicKV can provide up to 16x speedup in attention computation, 4.8x lower decode latency, and 7.3x higher throughput, all while maintaining a minimal accuracy drop of just 1.76%.
Achieving up to 16x faster attention computation with only a 1.76% accuracy loss could revolutionize the deployment of long-context LLMs in real-time applications.
Long-context LLM services now sustain prompts with hundreds of thousands to millions of tokens, making the key-value (KV) cache a first-order serving cost. Because the cache grows linearly with context length, it can exhaust GPU memory, force smaller batches, and reduce serving throughput. Prior KV cache compression techniques typically target only the sequence dimension or only the channel dimension, which leaves limited headroom as context windows scale. Compressing both dimensions promises higher memory reduction, but applying the two forms of compression directly leads to significant accuracy loss. This paper introduces MosaicKV, a dynamic two-D (dimensional) KV cache compression system for extremely long-context serving. MosaicKV uses dynamic two-D compression to address the accuracy challenge, exploiting the non-uniform importance distribution of elements within the KV cache. Instead of applying one compression pattern globally, MosaicKV identifies important elements for each KV vector and selects compression strategies at the granularity of KV cache segments. To address the performance challenge, where fine-grained sparsity and compression management overhead can offset the gains from compression, MosaicKV introduces compressed KV cache management. This mechanism uses underutilized GPU and CPU resources to maintain compressed KV caches and accelerate attention computation. Evaluation on an H800 GPU with multiple LLMs shows that MosaicKV delivers up to 16x attention speedup, 4.8x lower decode latency, and 7.3x higher throughput than the uncompressed baseline. At the same time, it reduces memory usage by 3x and incurs only 1.76% average accuracy loss on LongBench and RULER.
