Search papers, labs, and topics across Lattice.
This paper introduces ResonatorLM, a novel mechanism that substitutes traditional self-attention in transformers with a physics-inspired approach using causal functions of damped resonators to model long-context sequences. The method significantly improves training and decoding efficiency, achieving a remarkable 6.47x speedup at 32K tokens while also enhancing accuracy on long-context tasks, such as WikiText. These findings highlight the potential of ResonatorLM to overcome the limitations of conventional architectures in handling extensive contexts efficiently.
Achieving a 6.47x decoding speedup while improving accuracy on long-context tasks could redefine efficiency benchmarks in language modeling.
Contemporary language models are dominated by the transformer architecture, which leverages self-attention mechanisms to enable more efficient, parallelized training across a wide set of documents and corpora. This has allowed transformers to effectively model data across a wide range of modalities and contexts. However, transformers, along with their conventional counterparts such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs), often struggle to maintain efficiency when processing long contexts. We introduce ResonatorLM, a new mechanism that replaces attention with a physics-derived alternative. ResonatorLM treats token sequences as a single, driven one-dimensional latent field and replaces attention dot products with causal functions of damped resonators. We implement ResonatorLM on a traditional network architecture and test it on standard long-context modeling tasks. We find that in a small, 6M matched setting, training and prefill speedups increase with sequence length, decode speed reaches 6.47x compared to that of a standard, optimized transformer at 32K tokens, and accuracy reaches 61.31 percent (compared to 55.32 percent) on WikiText.
