Search papers, labs, and topics across Lattice.
This paper introduces adversarial decoys, optimized image patches that redirect attention away from true adversarial regions in Vision Transformers (ViTs), thereby enhancing the effectiveness of localized adversarial attacks. By decoupling the objectives of misclassification and defense evasion, the method allows for independent optimization of attention manipulation and adversarial effectiveness. Experiments demonstrate that decoys can significantly misdirect attention-based defenses while maintaining the integrity of the original attack, highlighting a critical flaw in relying solely on attention scores for adversarial detection.
Adversarial decoys can effectively misdirect attention-based defenses, revealing that attention magnitude is not a reliable indicator of adversarial relevance.
Vision Transformers (ViTs) remain vulnerable to localized adversarial attacks, e.g., adversarial patches, while recent test-time defenses mitigate them by suppressing image tokens with abnormally high attention scores. These defenses exploit a strong coupling between attention and adversarial effectiveness: adversarial tokens often need to attract substantial attention to influence the prediction. We introduce adversarial decoys, independently optimized image patches that redirect the attention, and therefore related defenses, toward selected target tokens. Rather than jointly optimizing misclassifications and defense evasion, our approach decouples the two objectives: the original adversarial region induces the incorrect prediction, while a separate decoy manipulates the attention ranking used by the defense. A layer-wise objective increases target-token attention and promotes these tokens above competing non-target ones. Since the decoy is optimized independently of the underlying attack, the method is attack-agnostic and can be easily integrated with any existing adversarial patch attack. Experiments on ImageNet across multiple ViT architectures and attacks show that decoys can redirect high attention scores away from the true adversarial region while preserving much of the attack effectiveness. These results reveal a fundamental limitation of using attention magnitude as an indicator of adversarial relevance.
