Search papers, labs, and topics across Lattice.
This paper introduces PASTA, a novel patch-agnostic backdoor attack on Vision Transformers (ViTs) that leverages the "Trigger Radiating Effect" (TRE) to activate backdoors across neighboring patches, even with triggers placed at arbitrary locations during inference. To enhance TRE, PASTA employs a multi-location trigger insertion strategy during training. The method formulates a bi-level optimization problem with an adaptive backdoor learning framework to maintain both visual and attention stealthiness while ensuring high attack success rates.
Forget single-location triggers: PASTA shows you can backdoor Vision Transformers with triggers placed *anywhere* and still achieve 99% attack success, all while being nearly undetectable.
Vision Transformers (ViTs) have achieved remarkable success across vision tasks, yet recent studies show they remain vulnerable to backdoor attacks. Existing patch-wise attacks typically assume a single fixed trigger location during inference to maximize trigger attention. However, they overlook the self-attention mechanism in ViTs, which captures long-range dependencies across patches. In this work, we observe that a patch-wise trigger can achieve high attack effectiveness when activating backdoors across neighboring patches, a phenomenon we term the Trigger Radiating Effect (TRE). We further find that inter-patch trigger insertion during training can synergistically enhance TRE compared to single-patch insertion. Prior ViT-specific attacks that maximize trigger attention often sacrifice visual and attention stealthiness, making them detectable. Based on these insights, we propose PASTA, a twofold stealthy patch-wise backdoor attack in both pixel and attention domains. PASTA enables backdoor activation when the trigger is placed at arbitrary patches during inference. To achieve this, we introduce a multi-location trigger insertion strategy to enhance TRE. However, preserving stealthiness while maintaining strong TRE is challenging, as TRE is weakened under stealthy constraints. We therefore formulate a bi-level optimization problem and propose an adaptive backdoor learning framework, where the model and trigger iteratively adapt to each other to avoid local optima. Extensive experiments show that PASTA achieves 99.13% attack success rate across arbitrary patches on average, while significantly improving visual and attention stealthiness (144.43x and 18.68x) and robustness (2.79x) against state-of-the-art ViT defenses across four datasets, outperforming CNN- and ViT-based baselines.
