Search papers, labs, and topics across Lattice.
This paper explores the limitations of traditional transformers in lifelong continual learning due to the quadratic nature of attention mechanisms, which restricts their ability to process long sequences. The authors advocate for the use of parametric forms of attention to enable transformers to manage a lifetime of context while adhering to fixed hardware constraints. Key findings reveal that while parametric attention can maintain a constant memory footprint, it still faces challenges related to memory capacity and the efficiency of online updates, prompting the authors to propose open questions for future research.
Parametric attention mechanisms could be the key to unlocking lifelong learning in transformers, but they still struggle with memory limitations and costly updates.
Lifelong continual learning remains an obstacle on the path to human-like intelligence. Modern transformers show sparks of intelligence with in-context learning. The quadratic nature of attention, however, prohibits transformers from performing this process on arbitrarily long sequences. In this work, we argue that extending in-context learning to lifelong settings is a practical solution for continual learning in AI agents. In particular, we argue that \emph{parametric forms of attention} are needed to understand a lifetime of context with transformers on a fixed hardware budget. These attention mechanisms learn the relationship between keys and their associated values at test-time with parametric regression. Our generalization of parametric approaches (linear attention, state-space models, fast weight programmers, and test-time training layers) contrasts with nonparametric counterparts like softmax attention. They replace the ever-growing key-value cache with an online-trainable neural network, maintaining a constant memory footprint. We highlight how parametric attention currently fall short of lifelong learning due to limited memory capacity or costly online updates. To address these issues, we pose a set of open questions with novel insights to guide the field toward long-horizon agents.
