Search papers, labs, and topics across Lattice.
StructKV addresses the KV cache bottleneck in long-context LLMs by selectively compressing the cache based on a novel structure-aware approach. It identifies and preserves "global information hubs" by aggregating attention patterns across all layers using a Global In-Degree Centrality metric, and it dynamically selects the optimal compression layer. Experiments on LongBench and RULER show that StructKV maintains performance while reducing memory footprint, improving long-range dependency handling and retrieval.
LLMs can maintain long-range dependencies and retrieval robustness in long contexts even with compressed KV caches, if you preserve the structural skeleton of information flow.
As Large Language Models (LLMs) scale to support context windows exceeding one million tokens, the linear growth of Key-Value (KV) cache imposes severe memory capacity and bandwidth bottlenecks, constraining the efficiency of long-context inference. Existing compression approaches typically prioritize tokens based on local saliency metrics to decouple prefill computation from decoding memory. However, these methods often rely on local saliency snapshots at a specific layer, thereby systematically discarding tokens that act as global information hubs across the network depth but appear temporarily dormant at the specific layer selected for pruning. To address this limitation, we propose StructKV, a structure-aware KV cache compression framework that introduces three core innovations: First, Global In-Degree Centrality aggregates attention patterns across the network depth to identify global information hubs. Second, Dynamic Pivot Detection utilizes information-theoretic metrics to adaptively locate the optimal layer for compression. Finally, Structural Propagation and Decoupling separates the computational budget from the memory storage budget. Experimental results on the LongBench and RULER benchmarks demonstrate that StructKV effectively preserves long-range dependencies and retrieval robustness.
