Search papers, labs, and topics across Lattice.
This paper introduces LongCat Sparse Attention (LSA), a novel framework designed to optimize long-context modeling by addressing the inefficiencies of the DeepSeek Sparse Attention's Lightning Indexer. By employing Streaming-Aware Indexing, Cross-Layer Indexing, and Hierarchical Indexing, LSA reduces the expensive $O(L^2)$ scoring overhead and enhances memory access patterns, enabling efficient processing of context lengths up to one million tokens. Extensive experiments show that LSA achieves performance comparable to full attention mechanisms across various benchmarks, paving the way for the development of the LongCat-2.0 model with 1.6 trillion parameters.
LSA slashes indexing overhead while maintaining full attention performance, enabling efficient long-context processing for models with up to one million tokens.
DeepSeek Sparse Attention (DSA) enables efficient long-context modeling through its Lightning Indexer. However, practical deployment remains constrained by the indexer's expensive $O(L^2)$ scoring overhead and the hardware-inefficient, discontinuous memory-access patterns induced by its outputs. To address these system-level bottlenecks, we introduce LongCat Sparse Attention (LSA), a hardware-algorithm co-designed framework comprising three complementary and orthogonal strategies: (1) Streaming-Aware Indexing, which selectively converts scattered KV entries into hardware-aligned contiguous layouts to enable coalesced HBM access; (2) Cross-Layer Indexing, which amortizes indexing overhead by reusing the results produced by a single layer across consecutive layers, supported by cross-layer distillation; and (3) Hierarchical Indexing, which adopts a coarse-to-fine scoring scheme to progressively narrow the candidate set for each query, thereby substantially reducing indexing computation. Extensive scaling experiments, ranging from 69B-A3B to 560B-A27B models, demonstrate that LSA consistently achieves performance on par with full attention across both general-purpose and long-context benchmarks. Moreover, LSA supports native training with context lengths of up to one million tokens and underpins the development of LongCat-2.0 (1.6T-A48B). To facilitate further research, we also introduce and open-source LongCat-Flash-Lite-Sparse (69B-A3B), which integrates LSA into LongCat-Flash-Lite and incorporates an updated long-context training corpus.
