Search papers, labs, and topics across Lattice.
This paper reveals a novel approach to backdooring vision-language models (VLMs) by allowing attackers to implant a programmable backdoor that can dynamically control target captions at inference time. By employing a heuristic poisoning strategy and feature-space trigger steganography, the method enables attackers to synthesize stealthy triggers for previously unseen target-caption pairs without the need for retraining the model. The results demonstrate high success rates for any-to-any caption control while maintaining the model's clean utility and resilience against traditional backdoor defenses.
A single poisoning phase can create a programmable backdoor in VLMs, enabling dynamic control over unseen target captions without retraining.
Existing vision-language model (VLM) backdoors are usually treated as static vulnerabilities: one-to-one and N-to-N attacks bind one or more triggers to a finite set of targets before victim training. This assumption substantially underestimates the threat. We show that a single poisoning phase can implant a programmable backdoor into a VLM, allowing an attacker to choose previously unseen target-caption semantics at inference time and synthesize corresponding stealthy triggers on demand. Unlike fixed-mapping attacks, the proposed any-to-any caption-control paradigm decouples post-training target selection from poisoning, enabling dynamic control of target captions without retraining the VLM. Our method has two components. First, a heuristic poisoning strategy exposes the model to diverse trigger-caption pairs, encouraging it to learn a general trigger-as-instruction rule rather than memorize a specific backdoor pattern. Second, a feature-space trigger steganography method maps any attacker-specified target caption to a stealthy visual trigger, implemented as either a norm-controlled perturbation or a non-semantic patch. Once inserted into arbitrary images, these triggers cause the poisoned VLM to generate outputs semantically aligned with the chosen target caption, even when the target was unseen during poisoning. Extensive experiments show that our attack achieves high any-to-any caption-control success rates, preserves clean model utility, and remains effective under several classical backdoor defenses.
