Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To overcome the noisy, interaction-isotropic prior of dense self-attention on structured data, this work introduces AXON, an architecture that factorizes attention into dedicated spatial (cross-channel) and temporal paths. Restricting token interactions along canonical physical axes suppresses irrelevant correlations that typically degrade representations in low-SNR domains like EEG and audio spectrograms. AXON consistently outperforms standard dense attention baselines across six EEG benchmark tasks under both linear probing and full fine-tuning, driven largely by learned, layer-dependent convex gating between the two paths.
Dense attention drowns low-SNR, multi-channel data in spurious cross-token noise; factorizing attention into orthogonal spatial and temporal paths with adaptive gating systematically outperforms isotropic Transformers on neural and audio signals.
Dense self-attention treats all token pairs as equally plausible before learning, an interaction-isotropic prior that can be mismatched to structured signals. For structured, low signal-to-noise ratio (SNR) signals such as EEG, dependencies are organized along the electrode and time axes, and this uniform prior exposes each token to many irrelevant interactions. We introduce Adaptive Anisotropic Attention (AAA), which splits attention into two paths: a temporal path, where each token attends to the tokens of its own electrode across time, and a spatial path, where it attends to the tokens of the other electrodes at the same time step. A small gate predicts, for every token, a convex combination of the two path outputs: two non-negative weights that sum to one. On six EEG downstream tasks, the resulting model, AXON (AXis-factorized Operator Network), improves mean balanced accuracy over a dense baseline under both linear probing and full fine-tuning. We show that both paths (temporal and spatial) are necessary and that the weighted sum beats a hard choice of one path; most of the benefit comes from the gate learning a different temporal/spatial balance at each layer of the network. Controlled audio spectrogram experiments show that axis factorization transfers beyond EEG. These results suggest that aligning attention with the natural axes of structured signals provides a useful inductive bias.