Search papers, labs, and topics across Lattice.
This paper introduces BinaryPC, a training-free hashing-based sparse attention mechanism designed to enhance the efficiency of long-context large language models (LLMs) during decoding. By leveraging binary principal components to create compact hash codes, BinaryPC effectively preserves structural information without the need for additional training or complex hashing techniques. Experimental results demonstrate that BinaryPC not only maintains accuracy comparable to full attention but also significantly boosts decoding throughput, achieving a 3.56脳 improvement over existing methods like FlashAttention.
Achieving a 3.56脳 increase in decoding throughput without sacrificing accuracy, BinaryPC revolutionizes efficiency in long-context LLMs.
Long-context large language models (LLMs) are increasingly deployed in real-world applications, yet self-attention remains a major efficiency bottleneck -- especially during decoding -- due to the necessity of repeatedly processing ever-growing key-value (KV) caches. Existing sparse attention reduce computation by attending to fewer KV pairs, but often suffer from substantial accuracy degradation, require additional training, or rely on expensive hashing. In this work, we present BinaryPC, a training-free, data-aware hashing-based sparse attention for long-context LLMs. BinaryPC constructs compact binary hash codes and corresponding hash function by computing binary principal components of data. Unlike Locality-Sensitive Hashing (LSH) with data-independent random projections or learned non-linear hashing methods, BinaryPC constructs binary codes that explicitly preserve the structural information of data without requiring gradient-based training. Comprehensive experiments across multiple model families and long-context benchmarks show that BinaryPC preserves accuracy relative to full attention while achieving superior performance among sparse and hashing-based baselines. On modern GPUs, BinaryPC improves end-to-end decoding throughput by 3.56$\times$ over the FlashAttention kernel. Our code is available at https://github.com/yudaohai666/BPC.
