Search papers, labs, and topics across Lattice.
The paper introduces Multi-turn Adaptive Prompting Attack (MAPA), a novel jailbreak attack against large vision-language models (LVLMs) that alternates text and vision attack actions at each turn and refines the attack trajectory across turns. MAPA addresses the vulnerability of existing multi-turn jailbreaks to visual inputs that trigger safety defenses in LVLMs. The proposed method achieves significant improvements in attack success rates (11-35%) compared to state-of-the-art methods on benchmarks including LLaVA-V1.6-Mistral-7B, Qwen2.5-VL-7B-Instruct, Llama-3.2-Vision-11B-Instruct and GPT-4o-mini.
LVLMs are more vulnerable than you think: a carefully crafted sequence of alternating text and visual prompts can bypass their safety mechanisms with significantly higher success rates.
Multi-turn jailbreak attacks are effective against text-only large language models (LLMs) by gradually introducing malicious content across turns. When extended to large vision-language models (LVLMs), we find that naively adding visual inputs can cause existing multi-turn jailbreaks to be easily defended. For example, overly malicious visual input will easily trigger the defense mechanism of safety-aligned LVLMs, making the response more conservative. To address this, we propose MAPA: a multi-turn adaptive prompting attack that 1) at each turn, alternates text-vision attack actions to elicit the most malicious response; and 2) across turns, adjusts the attack trajectory through iterative back-and-forth refinement to gradually amplify response maliciousness. This two-level design enables MAPA to consistently outperform state-of-the-art methods, improving attack success rates by 11-35% on recent benchmarks against LLaVA-V1.6-Mistral-7B, Qwen2.5-VL-7B-Instruct, Llama-3.2-Vision-11B-Instruct and GPT-4o-mini.
