Search papers, labs, and topics across Lattice.
This paper introduces PhaseWin, an efficient search algorithm designed to enhance faithful visual attribution in vision and vision-language models by reformulating the greedy selection process into a phased window-search approach. By alternating between global candidate screening, adaptive pruning, and localized window refinement, PhaseWin significantly reduces the evaluation complexity from quadratic to linear while maintaining high levels of faithfulness in attribution. Extensive experiments across various tasks, including image classification and object detection, demonstrate that PhaseWin achieves superior performance with fewer model evaluations compared to existing methods.
PhaseWin slashes the computational cost of visual attribution from quadratic to linear while preserving high faithfulness, revolutionizing how we interpret model decisions.
Visual attribution is a fundamental tool for interpreting modern vision and vision-language models, particularly when their decisions must be inspected, diagnosed, or audited. Its goal is to explain how a model's decision depends on local regions of the visual input, typically by assigning an importance ordering over candidate image regions. Given an image partitioned into $n$ regions, faithful attribution can be cast as an ordered subset-search problem, in which progressively inserting the selected regions should recover the target model response as early as possible. Exhaustive search over region subsets incurs exponential cost, while the widely used greedy search still requires a quadratic number of model evaluations, because every selection step rescores all remaining candidates. We propose PhaseWin, an efficient subset-search algorithm for faithful visual attribution. PhaseWin reorganizes greedy region selection into a phased window-search procedure: rather than re-evaluating the full candidate set at every step, it alternates between global candidate screening, adaptive pruning, and localized window refinement, while preserving the essential region-ranking behavior of greedy search. We analyze PhaseWin under monotone evidence-accumulation conditions and show that, under feature-level structural assumptions, it attains controllable linear evaluation complexity together with near-greedy faithfulness guarantees. Extensive experiments on image classification, object detection, visual grounding, and image captioning show that, among all compared attribution methods, PhaseWin reaches high faithfulness with the fewest forward passes, empirically realizing the predicted reduction from $O(n^2)$ to $O(n)$. The code is available at https://github.com/Qihuai27/phasewin-va.
