Search papers, labs, and topics across Lattice.
Triadic linear attention generalizes matrix-valued states in linear attention to third-order tensor states by using a triadic outer product for state updates and tensor contraction for state reads. This approach enables a parameter-efficient $E$-fold increase in state dimensionality, significantly enhancing long-context language modeling and recall performance compared to existing methods that enlarge state size. The proposed method is compatible with data-dependent forgetting and chunkwise-parallel training, demonstrating substantial improvements on benchmark tasks.
A simple triadic outer product expansion of linear attention's state dramatically boosts long-context modeling capacity without a proportional parameter increase.
Recurrent neural networks (RNNs) compress the historical context into a memory state of fixed size, thus allowing for constant-time inference. The memory state size is a crucial factor in their performance, as exemplified by the strong performance and resurgence of linear attention, which extends the vector-valued hidden states of ordinary RNNs to matrix-valued hidden states. Crucially, linear attention does so in a parameter-efficient way, in particular by using an outer product of the key and value vectors to write to the matrix-valued hidden state. We generalize this construction and propose triadic linear attention, which writes the triadic outer product of a key, a second key, and a value, into a third-order (i.e., 3D) tensor state, and reads from it by contracting both key axes with two queries. An $E$-dimensional second key thus yields an $E$-fold increase in state size while adding only two projections. Triadic linear attention is compatible with data-dependent forgetting, the delta rule, and chunkwise-parallel training. Applied to Gated DeltaNet and scalar-gated linear attention, triadic linear attention substantially improves long-context language modeling and recall, outperforming alternatives that enlarge the state.
MIT CSAIL