Search papers, labs, and topics across Lattice.
This paper introduces Denoising Attention (DnA), a novel approach that enhances multihead attention in visual perception tasks by separating relevant and irrelevant features through distinct subspaces. By employing positive and negative queries, DnA effectively reduces noise in attention patterns, leading to improved performance on various visual tasks. The method demonstrates a significant absolute gain of 0.8% on ImageNet-1K and further improvements in video understanding tasks, showcasing its versatility and effectiveness.
DnA achieves a notable 0.8% improvement on ImageNet-1K by effectively filtering out irrelevant features in attention mechanisms.
The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy attention patterns that dilute relevant features and degrade its performance. In this paper, we propose Denoising Attention or DnA, in which, first, a positive query identifies which image features belong to the correct class, and a negative query identifies closely associated but irrelevant image features. DnA then projects these interactions into two distinct subspaces with larger principal angles, promoting subspace separation and improved discriminability. Using a ViT-B backbone, our proposed DnA achieves an absolute gain of 0.8% on ImageNet-1K compared to the baseline. We further show improvements across multiple visual understanding tasks, including video understanding with video transformers (1.8%) and video LLMs (0.5%). Our extensive empirical analyses justify the design choices involving two interacting subspaces and the denoising effect of DnA.
