Search papers, labs, and topics across Lattice.
HybridKV is introduced as a novel framework to compress the KV cache in multimodal large language models (MLLMs) by classifying attention heads into static or dynamic types based on text-centric attention. It then employs a hierarchical budget allocation scheme and compresses static heads using text-prior pruning and dynamic heads via chunk-wise retrieval. Experiments on Qwen2.5-VL-7B across 11 multimodal benchmarks demonstrate a 7.9x reduction in KV cache memory and a 1.52x speedup in decoding, with minimal or no performance degradation.
MLLMs can achieve up to 7.9x KV cache compression and 1.52x faster decoding without sacrificing performance by intelligently compressing different attention heads with distinct strategies.
Multimodal Large Language Models (MLLMs) have advanced unified reasoning over text, images, and videos, but their inference is hindered by the rapid growth of key-value (KV) caches. Each visual input expands into thousands of tokens, causing caches to scale linearly with context length and remain resident in GPU memory throughout decoding, which leads to prohibitive memory overhead and latency even on high-end GPUs. A common solution is to compress caches under a fixed allocated budget at different granularities: token-level uniformly discards less important tokens, layer-level varies retention across layers, and head-level redistributes budgets across heads. Yet these approaches stop at allocation and overlook the heterogeneous behaviors of attention heads that require distinct compression strategies. We propose HybridKV, a hybrid KV cache compression framework that integrates complementary strategies in three stages: heads are first classified into static or dynamic types using text-centric attention; then a top-down budget allocation scheme hierarchically assigns KV budgets; finally, static heads are compressed by text-prior pruning and dynamic heads by chunk-wise retrieval. Experiments on 11 multimodal benchmarks with Qwen2.5-VL-7B show that HybridKV reduces KV cache memory by up to $7.9\times$ and achieves $1.52\times$ faster decoding, with almost no performance drop or even higher relative to the full-cache MLLM.
