Search papers, labs, and topics across Lattice.
This paper introduces a dual adversarial fine-tuning framework designed to enhance the robustness of Large Vision-Language Models (LVLMs) against adversarial attacks by integrating visual and semantic supervision from two distinct modalities. By employing a visual supervision branch that utilizes features from clean images and a semantic supervision branch that maintains caption-image alignment, the framework effectively generalizes across multiple downstream tasks without requiring separate task-specific retraining. Experimental results show that this approach significantly outperforms existing state-of-the-art methods in adversarial robustness across various tasks, including zero-shot classification, image captioning, and visual question answering (VQA).
Adversarial attacks can be effectively mitigated in LVLMs by jointly optimizing visual and semantic supervision, leading to enhanced robustness across diverse tasks.
While Large Vision-Language Models (LVLMs), represented by LLaVA and GPT-4V, have demonstrated remarkable capabilities, their visual inputs remain vulnerable to adversarial attacks, posing significant security risks. Existing defense methods predominantly target single-task scenarios (e.g., zero-shot classification) and consequently lack generalizability across various multimodal tasks. To address this limitation, we propose a dual adversarial fine-tuning framework that jointly optimizes visual and semantic supervision signals from two modalities, enhancing model robustness while generalizing across multiple downstream tasks. The proposed framework comprises two core components, i.e., $\textbf{Visual}$ supervision branch and $\textbf{Semantic}$ supervision branch. The former branch leverages features from clean images, extracted via a frozen original vision encoder, to guide adversarial robustness while the latter incorporates caption-image alignment as a contextual signal to preserve semantic coherence under attack. Moreover, our method achieves cross-task robustness by simply replacing the CLIP vision encoder in the original model, with no need of separate task-specific retraining or architecture modifications.Extensive experiments demonstrate that our approach outperforms the state-of-the-art method in adversarial robustness evaluation across zero-shot classification, image captioning, and visual question answering (VQA) tasks.
