Search papers, labs, and topics across Lattice.
This paper introduces RaBitQCache, a novel sparse attention framework that leverages randomized rotated binary quantization and binary-INT4 arithmetic to optimize Key-Value (KV) cache efficiency in long-context Large Language Model inference. By employing an unbiased proxy score with a proven error bound, the framework enables adaptive Top-p retrieval that adjusts the token budget based on actual attention sparsity, overcoming the limitations of existing methods. Evaluations reveal that RaBitQCache not only accelerates inference and reduces memory I/O but also maintains generation quality, outperforming state-of-the-art baselines.
RaBitQCache accelerates long-context LLM inference while cutting memory I/O by intelligently adapting token budgets based on attention sparsity.
Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased. To address these limitations, we propose RaBitQCache, a novel sparse attention framework that utilizes randomized rotated binary quantization and high-throughput binary-INT4 arithmetic to efficiently estimate attention weights. Our proxy score serves as an unbiased estimator with a proven error bound, enabling adaptive Top-p retrieval that dynamically adjusts the token budget based on actual attention sparsity. We further implement a hardware-aware system with asynchronous pipelining and lazy updates to mask overhead. Evaluations demonstrate that RaBitQCache significantly accelerates inference and reduces memory I/O while preserving generation quality compared to state-of-the-art baselines. Code is available at https://github.com/Sakuraaa0/RaBitQCache.git.
