Search papers, labs, and topics across Lattice.
This paper introduces FourierQK, a novel approach that applies FFT-based spectral preprocessing to query-key projections in transformers, significantly enhancing attention mechanisms for character-level language modeling. The method achieves substantial performance improvements on the TinyShakespeare dataset, with a fixed random spectral filter yielding a validation score of 1.031 and learned frequencies achieving scores as low as 0.236, representing a 79% reduction over standard dot-product attention. The findings highlight the importance of global frequency-domain mixing in transformer architectures, distinguishing this approach from traditional methods and establishing a new architectural boundary for spectral attention.
Spectral preprocessing of query-key projections can reduce attention computation costs by up to 79% while preserving performance in character-level language modeling.
FFT-based spectral preprocessing of learned query-key (Q/K) projections substantially improves transformer attention on character-level language modelling. On TinyShakespeare: a fixed random spectral filter achieves val=1.031 (Delta=+0.443); a single learned frequency at paragraph scale achieves val=0.608 (Delta=+0.867); and four learned frequencies spanning paragraph to word scale achieve val=0.309 (Delta=+1.166), a 79% reduction over standard dot-product attention. The single-frequency result is confirmed across three random seeds (mean val=0.236, std=0.019). The four frequencies converge to a near-geometric multi-scale ordering (49, 27, 10, 6 tokens/cycle) corresponding to paragraph, sub-paragraph, phrase, and word scales. The gain is specific to spectral preprocessing: random orthogonal and non-orthogonal projections of Q/K produce no measurable improvement, suggesting the benefit comes from global frequency-domain mixing rather than metric distortion. All results are verified by a shuffled-validation diagnostic against positional leakage. Causal filters (Gaussian, Mexican Hat, Morlet) do not improve over standard attention at character-level tokenisation: the bilateral FFT kernel is structurally non-causal, coupling every position to future tokens. This defines an architectural boundary between bilateral spectral attention (this paper) and genuinely causal spectral attention at word-scale tokenisation (companion paper MorletQK). This work is architecturally distinct from FNet (Lee-Thorp et al., 2021), which replaces attention with Fourier mixing of token embeddings. Here, spectral preprocessing applies only to Q/K projections while the full attention score structure is preserved.
