Search papers, labs, and topics across Lattice.
This paper introduces GhostPrompt, a novel adversarial prompting technique designed to enhance the transferability of adversarial prompts across different images in Vision-Language Models (VLMs). By employing a joint optimization strategy that focuses on generating image-invariant adversarial features, GhostPrompt significantly improves attack success rates by over 30% compared to existing state-of-the-art methods while also reducing computational time by approximately 70%. This advancement addresses the limitations of traditional text-based adversarial attacks, which are typically constrained to specific images, thereby broadening the applicability of adversarial prompting in VLMs.
GhostPrompt achieves over 30% higher attack success rates while slashing computation time by nearly 70%, transforming how adversarial prompts can be utilized across diverse images.
Vision-Language Models (VLMs) are known to be vulnerable to adversarial attacks, where subtle perturbations to images or texts induce erroneous outputs. However, most text-based attacks are adapted from language-model-centric methods, in which the visual input is fixed during optimization, resulting in adversarial prompts that are tied to specific images and thus limiting their attack effectiveness. To this end, we first introduce a new research perspective: cross-image transferability for adversarial prompts. We then propose GhostPrompt, an adversarial prompt that is optimized once and reused to steer VLM outputs toward attacker-specified responses across diverse images. GhostPrompt employs a joint optimization that distills image-invariant adversarial features into the prompt by"worst-case"generation. Specifically, it alternates between constructing hard visual conditions for the current prompt and updating the prompt to remain effective under these conditions. Extensive experiments on prevalent VLMs verify that \ourmethod achieves an improvement of over 30% in attack success rates compared to state-of-the-art (SoTA) baselines, while reducing computation time by ~70%. Our code is avalable at https://github.com/Ye-ze-yu/GhostPrompt.
