Search papers, labs, and topics across Lattice.
The paper introduces Semantic-Augmented Dynamic Contrastive Attack (SADCA), a novel method for generating transferable adversarial examples against vision-language pre-training (VLP) models. SADCA dynamically disrupts cross-modal alignment by establishing a contrastive learning mechanism between adversarial, positive, and negative samples, thereby reinforcing semantic inconsistency. Experiments demonstrate that SADCA significantly improves adversarial transferability across multiple datasets and models compared to existing attack methods by also leveraging input transformations for semantic augmentation.
Existing vision-language attacks are weak, but this new method, SADCA, uses dynamic contrastive learning and semantic augmentation to create adversarial examples that transfer much more effectively across models and datasets.
With the rapid advancement and widespread application of vision-language pre-training (VLP) models, their vulnerability to adversarial attacks has become a critical concern. In general, the adversarial examples can typically be designed to exhibit transferable power, attacking not only different models but also across diverse tasks. However, existing attacks on language-vision models mainly rely on static cross-modal interactions and focus solely on disrupting positive image-text pairs, resulting in limited cross-modal disruption and poor transferability. To address this issue, we propose a Semantic-Augmented Dynamic Contrastive Attack (SADCA) that enhances adversarial transferability through progressive and semantically guided perturbation. SADCA progressively disrupts cross-modal alignment through dynamic interactions between adversarial images and texts. This is accomplished by SADCA establishing a contrastive learning mechanism involving adversarial, positive and negative samples, to reinforce the semantic inconsistency of the obtained perturbations. Moreover, we empirically find that input transformations commonly used in traditional transfer-based attacks also benefit VLPs, which motivates a semantic augmentation module that increases the diversity and generalization of adversarial examples. Extensive experiments on multiple datasets and models demonstrate that SADCA significantly improves adversarial transferability and consistently surpasses state-of-the-art methods. The code is released at https://github.com/LiYuanBoJNU/SADCA.
