Search papers, labs, and topics across Lattice.
NestedKV, a training-free KV cache compression method, addresses the limitations of single-signal token ranking by incorporating global, block-level, and sliding-window key anchors. It scores tokens using multi-time-scale cosine anomaly and combines these rankings with a training-free outer learner via head-adaptive mixing and surprise-gated token routing. Experiments across various long-context benchmarks and models (Qwen3, Llama-3.2) demonstrate that NestedKV outperforms existing methods like KeyDiff, especially when retaining a small cache, achieving significant improvements in performance.
Forget brittle heuristics – NestedKV's multi-timescale anomaly detection and surprise-gated routing lets you compress KV caches far more aggressively without sacrificing long-context performance.
Long-context language models are limited by the memory footprint of the key-value (KV) cache. Existing training-free KV compression methods usually rank tokens by one importance signal -- attention, recency, layer-wise allocation, or key distinctiveness -- which becomes brittle when useful context is globally distinctive, locally episodic, or immediately relevant. We introduce NestedKV, a key-only KV cache compression method inspired by the Continuum Memory System in Nested Learning. NestedKV maintains global, block-level, and sliding-window key anchors, scores tokens by multi-time-scale cosine anomaly, and combines the resulting rankings with a training-free outer learner using head-adaptive mixing and surprise-gated token routing. The score is paired with adaptive per-head budgets and requires no training or LLM modification. Across RULER (4k--32k), LooGLE, LongBench, LongBench-E, InfiniteBench, and MMLU-Pro on Qwen3 and Llama-3.2 models, NestedKV is strongest when the retained cache is small. On Qwen3-4B, it improves over KeyDiff by up to 19.10 points on RULER and 19.29 on LongBench at $r=0.75$; at $r=0.95$, it retains 37.32 on LongBench versus 17.55 for KeyDiff.
