Search papers, labs, and topics across Lattice.
This paper introduces Inhibited Self-Attention (ISA), a novel mechanism for Vision Transformers that incorporates inhibitory signals to enhance feature selectivity and mitigate the influence of spurious correlations. By retaining negative attention scores, ISA sharpens focus on relevant objects while suppressing distractions from background features, leading to improved object-centric performance. Experiments on datasets such as ImageNet-1k and COCO reveal that ISA not only enhances selectivity but also boosts out-of-distribution generalization, confirming its efficacy in real-world applications.
Inhibited Self-Attention sharpens focus on relevant objects in Vision Transformers, significantly improving performance by reducing reliance on spurious correlations.
Vision Transformers (ViTs) have demonstrated remarkable performance in computer vision tasks. However, their self-attention mechanism often diffuses focus across background regions, relying on spurious correlations rather than object-relevant cues. Inspired by inhibitory mechanisms observed in biological vision systems, we propose the Inhibited Self-Attention (ISA), a novel self-attention that integrates inhibitory signals to enhance feature selectivity and suppress spurious responses. In contrast to conventional self-attention, which relies solely on positive attention values due to softmax normalization, our approach retains and utilizes negative attention scores to suppress irrelevant features and sharpen focus on objects of interest. Experiments across multiple datasets, including ImageNet-1k and COCO, and several robustness benchmarks demonstrate that ISA enhances object-centric selectivity, reduces shortcut reliance, and improves out-of-distribution generalization. Our analysis of relevance maps confirms that ViTs with ISA exhibit sharper, more localized focus on object-relevant regions while reducing distractions from non-relevant (background) features, enabling more reliable models. We release our code at https://github.com/prdvanderwal/inhibited-self-attention
