Search papers, labs, and topics across Lattice.
This paper introduces a novel learning-based visual attribution method that directly optimizes deletion and insertion metrics, bypassing surrogate objectives or heuristic teachers. They address the non-differentiability of ranking by framing the problem as permutation learning and employing a Gumbel-Sinkhorn relaxation for differentiable sorting. The resulting method achieves state-of-the-art performance in generating dense, pixel-level attribution maps, especially for vision transformers, with improved quantitative metrics and sharper, boundary-aligned explanations.
Get pixel-perfect attribution maps for vision transformers without the computational cost of perturbation or the biases of propagation methods.
Interpreting the decisions of complex computer vision models is crucial to establish trust and accountability, especially in safety-critical domains. An established approach to interpretability is generating visual attribution maps that highlight regions of the input most relevant to the model's prediction. However, existing methods face a three-way trade-off. Propagation-based approaches are efficient, but they can be biased and architecture-specific. Meanwhile, perturbation-based methods are causally grounded, yet they are expensive and for vision transformers often yield coarse, patch-level explanations. Learning-based explainers are fast but usually optimize surrogate objectives or distill from heuristic teachers. We propose a learning scheme that instead optimizes deletion and insertion metrics directly. Since these metrics depend on non-differentiable sorting and ranking, we frame them as permutation learning and replace the hard sorting with a differentiable relaxation using Gumbel-Sinkhorn. This enables end-to-end training through attribution-guided perturbations of the target model. During inference, our method produces dense, pixel-level attributions in a single forward pass with optional, few-step gradient refinement. Our experiments demonstrate consistent quantitative improvements and sharper, boundary-aligned explanations, particularly for transformer-based vision models.
