Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces cosine-normalized attention for hyperspectral image classification, arguing that dot-product attention mixes feature magnitude and orientation, which is suboptimal for hyperspectral data. They project query and key embeddings onto a unit hypersphere and apply a squared cosine similarity to emphasize angular relationships. Experiments on benchmark datasets demonstrate that the proposed approach outperforms several recent Transformer- and Mamba-based models under limited supervision, suggesting cosine-based scoring provides a reliable inductive bias.
Cosine similarity beats dot-product attention for hyperspectral image classification, offering a more reliable inductive bias and better performance, especially under limited supervision.
Transformer-based methods have improved hyperspectral image classification (HSIC) by modeling long-range spatial-spectral dependencies; however, their attention mechanisms typically rely on dot-product similarity, which mixes feature magnitude and orientation and may be suboptimal for hyperspectral data. This work revisits attention scoring from a geometric perspective and introduces a cosine-normalized attention formulation that aligns similarity computation with the angular structure of hyperspectral signatures. By projecting query and key embeddings onto a unit hypersphere and applying a squared cosine similarity, the proposed method emphasizes angular relationships while reducing sensitivity to magnitude variations. The formulation is integrated into a spatial-spectral Transformer and evaluated under extremely limited supervision. Experiments on three benchmark datasets demonstrate that the proposed approach consistently achieves higher performance, outperforming several recent Transformer- and Mamba-based models despite using a lightweight backbone. In addition, a controlled analysis of multiple attention score functions shows that cosine-based scoring provides a reliable inductive bias for hyperspectral representation learning.