Search papers, labs, and topics across Lattice.
This paper introduces ELSAA, an innovative method that efficiently approximates the attention mechanism in Transformers by combining low-rank and sparse representations without decomposing the learned projection matrices. By utilizing a dual-branch approach, ELSAA captures both high-similarity interactions and diffuse global interactions, while a denominator-aware fusion term ensures effective scaling between the two branches. The results demonstrate that ELSAA enables longer-context training while maintaining the quality of token-level interactions and broader contextual mixing, addressing a critical limitation in current Transformer architectures.
ELSAA achieves efficient attention approximation, allowing Transformers to handle longer inputs without sacrificing interaction quality.
The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths. Existing efficient attention methods usually reduce this bottleneck by either imposing sparsity, so that each query attends to only a small subset of keys, or by using low-rank/kernel sketches, so that global interactions are compressed into a lower-dimensional representation. We propose \emph{ELSAA}, an efficient low-rank and sparse approximation of attention. Importantly, ELSAA does \emph{not} decompose the learned projection or output matrices of the Transformer into sparse and low-rank factors. Instead, after dense projections produce $Q,K,V$, ELSAA approximates the induced attention score operator itself: a sparse branch captures selected high-similarity interactions, while a low-rank branch summarizes diffuse global interactions. Since the two branches can be normalized over supports with very different denominator mass, ELSAA introduces a denominator-aware fusion term that scales the sparse branch according to its estimated attention mass relative to the low-rank branch. This gives a practical framework for constructing low-rank and sparse attention outputs without materializing the full quadratic score matrix, aiming to enable longer-context training while preserving both sharp token-level interactions and broad contextual mixing.
