Search papers, labs, and topics across Lattice.
This paper introduces Decision-Aware Attention Propagation (DAP), an innovative method for enhancing the interpretability of Vision Transformers (ViTs) by integrating decision-relevant priors into the attention propagation process. By combining gradient-based localization with layer-wise attention rollout, DAP effectively captures both the structural flow of attention and the most relevant evidence for final predictions, resulting in more class-sensitive and faithful attribution maps. Experimental results across various ViT models demonstrate that DAP significantly outperforms existing attention-based explanation methods in both quantitative metrics and qualitative assessments, underscoring its potential for improving model interpretability.
DAP transforms how we interpret Vision Transformers by producing attribution maps that are not only more faithful but also significantly more class-sensitive than traditional methods.
Vision Transformers (ViTs) have become a dominant architecture in computer vision, yet their prediction process remains difficult to interpret because information is propagated through complex interactions across layers and attention heads. Existing attention based explanation methods provide an intuitive way to trace information flow. However, they rely mainly on raw attention weights, which do not explicitly reflect the final decision and often lead to explanations with limited class discriminability. In contrast, gradient based localization methods are more effective at highlighting class specific evidence, but they do not fully exploit the hierarchical attention propagation mechanism of transformers. To address this limitation, we propose Decision-Aware Attention Propagation (DAP), an attribution method that injects decision-relevant priors into transformer attention propagation. By estimating token importance through gradient based localization and integrating it into layer wise attention rollout, the method captures both the structural flow of attention and the evidence most relevant to the final prediction. Consequently, DAP produces attribution maps that are more class sensitive, compact, and faithful than those generated by conventional attention based methods. Extensive experiments across Vision Transformer variants of different model scales show that DAP consistently outperforms existing baselines in both quantitative metrics and qualitative visualizations, indicating that decision aware propagation is an effective direction for improving ViT interpretability.
