Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SISA (SSM-Informed Softmax Attention), a novel hybrid attention mechanism that integrates state space models (SSMs) directly into the attention computation by incorporating an SSM-derived importance term into the attention scores. By doing so, SISA achieves significantly improved performance on the LAMBADA benchmark, reaching a greedy score of 17.3%, surpassing both traditional Transformers and existing hybrid models like Mamba-3. The method also demonstrates a remarkable 7x faster retrieval convergence while maintaining perfect NIAH from the first step, establishing a new paradigm for SSM-attention hybrids through score-level fusion.
SISA achieves a 17.3% LAMBADA score by seamlessly integrating importance signals into attention, outperforming traditional models and redefining hybrid architecture.
Combining attention's global retrieval with the sequential importance signal of state space models (SSMs) is the open challenge of hybrid language modeling. Transformers see everywhere but cannot prioritize; SSMs know what matters but cannot revisit. Existing hybrids -- Jamba (block level) and Hymba (head level) -- place the two in separate compartments, so neither informs the other during the attention computation itself. We propose SISA (SSM-Informed Softmax Attention), which adds an SSM-derived importance term directly inside the attention score and realizes the full operation as a single SDPA call on augmented query/key vectors -- no recurrent state, no custom kernel. At 152M / 5B tokens, SISA reaches LAMBADA-greedy 17.3% (vs. Transformer 13.9 and Mamba-3 15.5) and attains NIAH 100% from step 1K, 7x faster than Transformer's retrieval convergence; at 369M, Mamba-3 leads LAMBADA while SISA preserves perfect NIAH and stock-SDPA execution. SISA thus defines a third design axis for SSM-attention hybrids -- score-level fusion -- beyond the block-level and head-level paradigms that have dominated the field.