Search papers, labs, and topics across Lattice.
Energy-Gated Attention (EGA) is introduced, which modulates value aggregation in Transformers based on the spectral energy of key token embeddings, learned via a single linear projection. This approach posits that informationally dense tokens concentrate spectral energy and should attract more attention. EGA achieves validation loss improvements of +0.103 on TinyShakespeare and +0.101 on Penn Treebank with minimal parameter overhead and no measurable computational cost, demonstrating the efficacy of spectrally-informed attention mechanisms.
Forget uniform token weighting – Transformers can achieve better performance by focusing attention on informationally dense tokens identified through spectral energy analysis.
Standard transformer attention computes pairwise similarity between queries and keys, treating all tokens as equally salient regardless of their intrinsic informational content. In turbulent fluid dynamics, coherent structures -- the energetically dominant, spatially organized patterns that persist amid background chaos -- carry a disproportionate fraction of total energy and govern all transport. We propose that tokens play an analogous role in transformer attention: informationally dense positions (morphological boundaries, syntactic heads, discourse markers) concentrate spectral energy and should attract proportionally more attention than background tokens (function words, repeated patterns, low-information filler). We propose Energy-Gated Attention (EGA): a simple modification that gates value aggregation by the spectral energy of key token embeddings, computed by a single learned linear projection that discovers the dominant spectral mode of the embedding field. On TinyShakespeare, EGA achieves +0.103 validation loss improvement with only 12,480 additional parameters (<0.26% overhead) and no measurable computational cost. The result is consistent on Penn Treebank (+0.101), demonstrating dataset independence. A systematic ablation across three wavelet families (fixed Morlet, Daubechies db2/db4, and a parametric Morlet) establishes that fixed structured bases are suboptimal -- the optimal energy direction is data-adaptive and non-sinusoidal -- while identifying learned wavelet packets as a promising open direction. The learned energy threshold converges to tau ~= 0.35 independently of initialization, corresponding to the fraction (~36%) of tokens carrying above-average spectral energy in English text, a stable linguistic property consistent with the fraction of content words in running English text.
