Search papers, labs, and topics across Lattice.
The paper introduces AsyncTLS, a two-level sparse attention mechanism that combines block-level filtering with token-level selection for efficient long-context LLM inference. To mitigate KV cache memory bottlenecks, AsyncTLS employs an asynchronous offloading engine that overlaps memory transfers with computation by exploiting temporal locality. Experiments on Qwen3 and GLM-4.7-Flash demonstrate that AsyncTLS achieves full attention accuracy while providing significant operator speedups (1.2x-10.0x) and end-to-end throughput improvements (1.3x-4.7x) on long contexts (48k-96k).
Achieve full-attention accuracy with 10x operator speedup and 4.7x throughput improvement in long-context LLM inference by overlapping KV cache transfers with computation.
Long-context inference in LLMs faces the dual challenges of quadratic attention complexity and prohibitive KV cache memory. While token-level sparse attention offers superior accuracy, its indexing overhead is costly; block-level methods improve efficiency but sacrifice precision. We propose AsyncTLS, a hierarchical sparse attention system that combines coarse-grained block filtering with fine-grained token selection to balance accuracy and efficiency, coupled with an asynchronous offloading engine that overlaps KV cache transfers with computation via temporal locality exploitation. Evaluated on Qwen3 and GLM-4.7-Flash across GQA, and MLA architectures, AsyncTLS achieves accuracy comparable to full attention while delivering 1.2x - 10.0x operator speedups and 1.3x - 4.7x end-to-end throughput improvements on 48k - 96k contexts.
