Search papers, labs, and topics across Lattice.
The paper introduces CHESS, a KV-cache management system for long-context LLMs that addresses the latency bottleneck caused by the growing KV cache size. CHESS employs a context-aware, hierarchical selection policy to dynamically reconstruct a coherent context relevant to the current decoding step, improving inference quality. By combining this algorithmic approach with a system-level design that minimizes data movement through coarse-grained selection, CHESS achieves significant speedups and maintains quality using only 1% of the KV cache, resulting in up to 4.56x higher throughput.
Forget brute-force KV cache: CHESS achieves comparable quality to full KV cache using only 1% of the memory, unlocking 4.56x higher throughput for long-context LLMs.
Long-context LLMs demand accurate inference at low latency, yet decoding becomes primarily constrained by KV cache as context grows. Prior pruning methods are largely context-agnostic: their token selection ignores step-wise relevance and local semantics, which undermines quality. Moreover, their irregular accesses and selection overheads yield only limited wall-clock speedups. To address this, we propose \textbf{CHESS}, an \textit{algorithm-system co-design} KV-cache management system. Algorithmically, CHESS introduces a context-aware, hierarchical selection policy that dynamically reconstructs a coherent context for the current decoding. System-wise, coarse granularity selection eliminates expensive data movement, fully realizing practical acceleration from theoretical sparsity. Extensive evaluations demonstrate that CHESS surpasses Full-KV quality using only \textbf{1\%} of the KV cache, delivers low-latency stable inference with up to \textbf{4.56$\times$} higher throughput, and consistently outperforms other strong baselines. Code is available at \href{https://anonymous.4open.science/r/CHESS-9958/}{https://anonymous.4open.science/r/CHESS/}.
