Search papers, labs, and topics across Lattice.
We introduce Random Feature Gaussian Process Attention (RFF-GPA), a novel module that approximates Gaussian Process attention with random Fourier features to achieve linear-time complexity. This approach enables principled uncertainty calibration in Transformers, a critical limitation for safety-critical applications, by efficiently computing the GP posterior mean and variance. Empirical results demonstrate that RFF-GPA maintains predictive accuracy while significantly improving calibration and reducing computational cost.
Transformer uncertainty is now reliably calibrated with linear-time attention.
Transformers provide a state-of-the-art modeling framework, yet poor calibration limits their reliability in safety-critical applications. A promising direction addresses this issue by interpreting attention as a Gaussian process (GP) posterior, which enables principled uncertainty calibration but incurs cubic complexity in sequence length due to the inversion of the kernel; although decoupled GP variants reduced the cost to quadratic, the computation remains prohibitive in practice. In this paper, we propose the plug-and-play random Fourier feature Gaussian process attention (RFF-GPA) module, which represents the attention as a GP with a stationary kernel approximated by random Fourier features. This low-rank approximation results in linear-time complexity for approximating the posterior mean and variance, making it far more scalable compared to previous work. Empirical results on multiple real-world datasets show that our attention module improves calibration while maintaining predictive accuracy, and simultaneously reduces computational complexity to linear in the sequence length.
