Search papers, labs, and topics across Lattice.
This study evaluates three subquadratic architectures鈥攛LSTM, Mamba-2, and Gated DeltaNet鈥攁cross tasks requiring complex dependencies, including code-model pre-training and time-series foundation model pre-training. The results reveal that xLSTM outperforms the other architectures, attributed to its superior state tracking and memory dynamics facilitated by a unique gating scheme. This work not only identifies xLSTM as the most effective model but also provides a unified framework to understand the architectural principles driving its performance advantages.
xLSTM outperforms its competitors in complex sequence modeling tasks by leveraging advanced state tracking and memory correction mechanisms.
Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost. Subquadratic architectures offer a scalable alternative. However, it remains unclear which designs yield the most effective sequence models. We compare three leading approaches: xLSTM, Mamba-2, and Gated DeltaNet. We evaluate these models on tasks with complex dependencies: (1) code-model pre-training, (2) distillation of code models from large language models, and (3) pre-training of time-series foundation models. Across these settings, xLSTM delivers the strongest overall performance. To explain xLSTM's advantage, we present a unified formulation and analyze the underlying architectural mechanisms, focusing on state tracking and memory dynamics. Our results show that xLSTM enables more flexible and stable memory correction via its gating scheme. We corroborate these findings on controlled synthetic length-generalization tasks. Overall, our findings indicate that xLSTM's gains on complex tasks stem from robust state tracking and accumulation.
