Search papers, labs, and topics across Lattice.
This paper introduces Functional Attention, a novel approach that reinterprets attention mechanisms in machine learning as functional correspondences between adaptive bases, moving beyond traditional token-wise attention methods. By replacing softmax affinities with structured linear operators, the method captures global dependencies in a resolution-invariant manner, leading to a more compact and generalizable representation. Experimental results show that Functional Attention achieves state-of-the-art performance across various operator learning tasks, including solving partial differential equations (PDEs), 3D segmentation, and regression, while maintaining robustness to different discretizations.
Functional Attention achieves state-of-the-art results in operator learning by transforming attention into a powerful tool for capturing global functional structures.
Learning mappings between infinite-dimensional function spaces, or operator learning, is essential for many machine learning applications. Although transformer-based operators are popular, they often rely on token-wise attention. These methods treat continuous fields as discrete tokens and usually ignore the global functional structure. We introduce \emph{Functional Attention}, which reinterprets attention as a functional correspondence between adaptive bases. Inspired by geometric functional maps, our method replaces softmax affinities with structured linear operators. This yields a compact, generalizable, resolution-invariant representation that explicitly captures global dependencies. Experiments demonstrate that \emph{Functional Attention} can match state-of-the-art performance in many operator learning tasks, including solving PDEs, 3D segmentation, and regression, while remaining robust to varying discretizations. Project page is available at https://github.com/xjffff/FUNCATTN.
