Search papers, labs, and topics across Lattice.
This paper investigates the vulnerability of Vision-Language Models (VLMs) to backdoor attacks and finds existing attacks are easily detectable using adapted defense techniques. To address this, they introduce Phantasia, a context-adaptive backdoor attack that generates poisoned outputs aligned with input semantics, enhancing stealth. Experiments across VLMs demonstrate Phantasia's superior attack success rate and stealth compared to existing methods, while maintaining benign performance.
VLMs are far more vulnerable to backdoor attacks than previously thought, and a new attack, Phantasia, makes them even harder to detect by dynamically tailoring malicious responses to fit the input context.
Recent advances in Vision-Language Models (VLMs) have greatly enhanced the integration of visual perception and linguistic reasoning, driving rapid progress in multimodal understanding. Despite these achievements, the security of VLMs, particularly their vulnerability to backdoor attacks, remains significantly underexplored. Existing backdoor attacks on VLMs are still in an early stage of development, with most current methods relying on generating poisoned responses that contain fixed, easily identifiable patterns. In this work, we make two key contributions. First, we demonstrate for the first time that the stealthiness of existing VLM backdoor attacks has been substantially overestimated. By adapting defense techniques originally designed for other domains (e.g., vision-only and text-only models), we show that several state-of-the-art attacks can be detected with surprising ease. Second, to address this gap, we introduce Phantasia, a context-adaptive backdoor attack that dynamically aligns its poisoned outputs with the semantics of each input. Instead of producing static poisoned patterns, Phantasia encourages models to generate contextually coherent yet malicious responses that remain plausible, thereby significantly improving stealth and adaptability. Extensive experiments across diverse VLM architectures reveal that Phantasia achieves state-of-the-art attack success rates while maintaining benign performance under various defensive settings.
