Search papers, labs, and topics across Lattice.
This paper analyzes the infinite-prompt limit of softmax attention applied to Gaussian mixtures, revealing its dual capability to perform linear operations when appropriate and engage in query-dependent selection for nonlinear tasks. By studying this regime, the authors demonstrate that softmax attention can effectively learn optimal solutions for statistical tasks like classification and denoising, outperforming linear attention on problems with latent structure. The findings underscore softmax attention's capacity to balance efficiency with sophisticated context exploitation.
Softmax attention selectively exploits latent structure in Gaussian mixtures, solving nonlinear tasks beyond linear attention's reach while matching linear performance on tractable problems.
Softmax attention, at the heart of Transformers, has demonstrated remarkable capabilities. Yet its underlying mechanisms remain only partially understood. Recent theoretical work studies Gaussian prompts, where the infinite-prompt limit reduces softmax attention to a linear map, but also removes the query-dependent selection that distinguishes it from linear attention. This work studies the infinite-prompt limit of softmax attention on Gaussian mixtures, which retain the tractability of Gaussian data while introducing latent structure, multimodality, and nonlinear dependencies. We show that softmax attention can represent and learn, via gradient-based methods, optimal solutions to a range of statistical tasks, including supervised classification and denoising. Our results highlight two complementary capabilities of softmax attention: it can recover linear tasks as effectively as its simpler linear counterpart, while also exploiting query-dependent context selection to solve nonlinear tasks beyond the reach of linear attention.
