Search papers, labs, and topics across Lattice.
This paper introduces Criticality-Aware Adversarial Training (CAAT), a method to efficiently adversarially train large Vision Transformers by fine-tuning only a small subset of parameters. CAAT identifies and focuses on the most robustness-critical parameters, using parameter-efficient fine-tuning (PEFT) when the number of critical parameters exceeds a threshold. Experiments on adversarial learning datasets show that CAAT achieves comparable robustness to full adversarial training while tuning only a small fraction of the parameters, outperforming other lightweight AT methods.
Adversarial training of large vision models doesn't have to break the bank: CAAT achieves comparable robustness to standard methods by tuning just 6% of the parameters.
Vision Transformer (ViT) models have achieved remarkable performance across various vision tasks, with scalability being a key advantage when applied to large datasets. This scalability enables ViT models to exhibit strong generalization capabilities. However, as the number of parameters increases, the robustness of ViT models to adversarial examples does not scale proportionally. Adversarial training (AT), one of the most effective methods for enhancing robustness, typically requires fine-tuning the entire model, leading to prohibitively high computational costs, especially for large ViT architectures. In this paper, we aim to robustly fine-tune only a small subset of parameters to achieve robustness comparable to standard AT. To accomplish this, we introduce Criticality-Aware Adversarial Training (CAAT), a novel method that adaptively allocates resources to the most robustness-critical parameters, fine-tuning only selected modules. Specifically, CAAT efficiently identifies parameters that contribute most to adversarial robustness. It then leverages parameter-efficient fine-tuning (PEFT) to robustly adjust weight matrices where the number of critical parameters exceeds a predefined threshold. CAAT exhibits favorable generalization when scaled to larger vision transformer architectures, potentially paving the way for adversarial training at scale, e.g, compared with plain adversarial training, CAAT incurs only a 4.3% decrease in adversarial robustness while tuning approximately 6% of its parameters. Extensive experiments on three widely used adversarial learning datasets demonstrate that CAAT outperforms state-of-the-art lightweight AT methods with fewer trainable parameters.
