Search papers, labs, and topics across Lattice.
This paper investigates the rank of adversarial perturbations for LLMs, finding that they reside in a low-dimensional subspace. Leveraging this insight, the authors propose a two-step approach to improve black-box adversarial attacks: first, project gradients into a low-dimensional subspace using a reference model and auxiliary data; second, confine the perturbation search to this subspace. Experiments across various models, datasets, and attack methods demonstrate that low-rank adversarial attacks achieve substantial improvements in efficiency and effectiveness compared to conventional methods.
Adversarial attacks on LLMs can be dramatically sped up and made more effective by exploiting the surprisingly low-rank structure of adversarial perturbations.
Low-Rank Adaptation (LoRA), which leverages the insight that model updates typically reside in a low-dimensional space, has significantly improved the training efficiency of Large Language Models (LLMs) by updating neural network layers using low-rank matrices. Since the generation of adversarial examples is an optimization process analogous to model training, this naturally raises the question: Do adversarial perturbations exhibit a similar low-rank structure? In this paper, we provide both theoretical analysis and extensive empirical investigation across various attack methods, model architectures, and datasets to show that adversarial perturbations indeed possess an inherently low-rank structure. This insight opens up new opportunities for improving both adversarial attacks and defenses. We mainly focus on leveraging this low-rank property to improve the efficiency and effectiveness of black-box adversarial attacks, which often suffer from excessive query requirements. Our method follows a two-step approach. First, we use a reference model and auxiliary data to guide the projection of gradients into a low-dimensional subspace. Next, we confine the perturbation search in black-box attacks to this low-rank subspace, significantly improving the efficiency and effectiveness of the adversarial attacks. We evaluated our approach across a range of attack methods, benchmark models, datasets, and threat models. The results demonstrate substantial and consistent improvements in the performance of our low-rank adversarial attacks compared to conventional methods.
