Search papers, labs, and topics across Lattice.
This survey investigates adversarial efficiency degradation attacks targeting input-adaptive optimizations in Vision Transformers (ViTs), specifically focusing on token pruning and early halting strategies. By analyzing two notable attacks, SlowFormer and DeSparsify, across three token-pruning frameworks, the study standardizes metrics such as GFLOPs and accuracy loss to assess the impact of these attacks on computational efficiency. The findings highlight the vulnerability of current optimization strategies and emphasize the need for lightweight countermeasures to protect against these attacks, particularly in low-power deployment scenarios.
Adversarial attacks can exploit input-adaptive optimizations in Vision Transformers, undermining their efficiency without sacrificing accuracy.
Vision Transformers (ViTs) increasingly rely on input-adaptive inference, such as token pruning and early halting, to meet energy and latency budgets. This survey examines a recent class of adversarial efficiency degradation attacks that target these mechanisms to increase computation without necessarily degrading accuracy. We unify and compare two representative attacks, SlowFormer (a universal adversarial patch) and DeSparsify (per-image perturbations), across three popular token-pruning frameworks: A-ViT, ATS, and AdaViT. We standardize reporting using GFLOPs, accuracy loss, and an Attack Success (AS) metric that measures how much of the model's compute savings the attack takes away. Understanding these attacks is crucial for designing countermeasures that not only mitigate risk but also remain lightweight, since deployment often occurs in low-power settings such as mobile or embedded devices. To organize our analysis, we focus on three questions: how input-adaptive optimizations (e.g., token pruning and early halting) create attack surfaces for efficiency degradation; how such attacks operate in practice and which optimizations are most vulnerable; and which defenses exist today and whether they meaningfully restore efficiency under attack.
