Search papers, labs, and topics across Lattice.
This paper provides a theoretical justification for continuous adversarial training (CAT) of LLMs by connecting it to in-context learning theory. They prove a robust generalization bound for linear transformers trained with embedding-space adversarial examples, showing that robustness is negatively correlated with the perturbation radius and related to the singular values of the embedding matrix. Based on this analysis, they introduce a regularization term based on singular values to improve CAT, demonstrating improved robustness-utility tradeoffs in real-world LLMs.
CAT's ability to defend against jailbreaks hinges on the singular values of the LLM's embedding matrix, offering a new handle for improving robustness.
Adversarial training (AT) is an effective defense for large language models (LLMs) against jailbreak attacks, but performing AT on LLMs is costly. To improve the efficiency of AT for LLMs, recent studies propose continuous AT (CAT) that searches for adversarial inputs within the continuous embedding space of LLMs during AT. While CAT has achieved empirical success, its underlying mechanism, i.e., why adversarial perturbations in the embedding space can help LLMs defend against jailbreak prompts synthesized in the input token space, remains unknown. This paper presents the first theoretical analysis of CAT on LLMs based on in-context learning (ICL) theory. For linear transformers trained with adversarial examples from the embedding space on in-context linear regression tasks, we prove a robust generalization bound that has a negative correlation with the perturbation radius in the embedding space. This clearly explains why CAT can defend against jailbreak prompts from the LLM's token space. Further, the robust bound shows that the robustness of an adversarially trained LLM is closely related to the singular values of its embedding matrix. Based on this, we propose to improve LLM CAT by introducing an additional regularization term, which depends on singular values of the LLM's embedding matrix, into the objective function of CAT. Experiments on real-world LLMs demonstrate that our method can help LLMs achieve a better jailbreak robustness-utility tradeoff. The code is available at https://github.com/fshp971/continuous-adv-icl.
