Search papers, labs, and topics across Lattice.
This paper introduces Multi-Modal Adversarial Synergy (MMAS), a black-box attack framework that crafts universal adversarial perturbations for both image and text inputs to compromise Large Vision-Language Models (LVLMs). MMAS uses wavelet-based texture constraints for imperceptible image perturbations and L-norm constraints for semantically coherent text perturbations, optimizing them jointly with a cross-modal regularization term. Experiments demonstrate that MMAS can effectively attack prevalent LVLMs, highlighting their vulnerability to multi-modal adversarial inputs.
LVLMs are surprisingly susceptible to universal, black-box adversarial attacks that synergistically combine imperceptible image perturbations with subtle text prompts.
Large Vision-Language Models (LVLMs) have transformed multi-modal understanding, excelling in tasks like image captioning and visual question answering by integrating visual and textual inputs. However, their robustness against adversarial attacks, particularly those exploiting both modalities, remains underexplored, posing risks to critical applications like autonomous driving and content moderation. Existing attacks focus on single modalities or require impractical white-box access, limiting their real-world relevance. In this paper, we introduce Multi-Modal Adversarial Synergy, a groundbreaking framework that crafts universal, black-box multi-modal attacks against LVLMs. MMAS simultaneously generates a texture scale-constrained universal adversarial perturbation for images and a learnable prompt perturbation for text, optimized jointly using only model queries. The image perturbation leverages wavelet-based texture constraints to ensure imperceptibility and robustness across diverse visual inputs. The text perturbation, constrained by an L-norm in the embedding space, maintains semantic coherence while steering outputs toward a target. A novel cross-modal regularization term aligns the perturbations' gradient directions, enhancing their synergistic impact and transferability across tasks and models. Extensive experiments show the strong universal adversarial capabilities of our proposed attack with prevalent LVLMs.
