Search papers, labs, and topics across Lattice.
MAC-Attention accelerates long-context LLM decoding by reusing prior attention computations for semantically similar recent queries, preserving fidelity and access to the full KV cache. It employs a match-amend-complete scheme involving pre-RoPE L2 matching, boundary rectification, and numerically stable fusion with fresh attention on the KV tail. Experiments on LongBench v2, RULER, and LongGenBench demonstrate up to 99% reduction in KV accesses, over 60% latency reduction at 128K context length, and up to 14.3x attention-phase speedups compared to FlashInfer, all while maintaining full-attention quality.
Achieve 14x attention speedups and 60% end-to-end latency reduction in long-context LLMs without sacrificing quality by reusing prior attention computations.
Long-context decoding in LLMs is IO-bound: each token re-reads an ever-growing KV cache. Prior accelerations cut bytes via compression, which lowers fidelity, or selection/eviction, which restricts what remains accessible, and both can degrade delayed recall and long-form generation. We introduce MAC-Attention, a fidelity- and access-preserving alternative that accelerates decoding by reusing prior attention computations for semantically similar recent queries. It starts with a match stage that performs pre-RoPE L2 matching over a short local window; an amend stage rectifies the reused attention by recomputing a small band near the match boundary; and a complete stage fuses the rectified results with fresh attention computed on the KV tail through a numerically stable merge. On a match hit, the compute and bandwidth complexity is constant regardless of context length. The method is model-agnostic and composes with IO-aware kernels, paged-KV managers, and MQA/GQA. Across LongBench v2 (120K), RULER (120K), and LongGenBench (16K continuous generation), compared to the latest FlashInfer library, MAC-Attention reduces KV accesses by up to 99%, cuts token generation latency by over 60% at 128K, and achieves over 14.3x attention-phase speedups, up to 2.6x end-to-end, while maintaining full-attention quality. By reusing computation, MAC-Attention delivers long-context inference that is both fast and faithful. Code is available here: https://github.com/YJHMITWEB/MAC-Attention.git
