Search papers, labs, and topics across Lattice.
This study reveals that pairing encoded jailbreak prompts with unrelated decoy images can significantly reduce the attack success rate (ASR) on Vision-Language Models (VLMs), highlighting a critical interaction in defense mechanisms. Through experiments across five advanced VLMs and various encoded-attack families, the researchers found that a caption-mediated defense (ECSO) can achieve up to a 73 percentage point drop in ASR when a decoy image is included. The findings suggest that the presence of an image, rather than its content, activates defense mechanisms, although the method's deployment raises concerns about benign refusal rates.
Pairing decoy images with encoded jailbreak prompts can reduce attack success rates by up to 73 percentage points, revealing a surprising interaction in VLM defense mechanisms.
We report a counter-intuitive interaction between image inputs and existing black-box defenses on Vision--Language Models (VLMs): pairing an encoded jailbreak prompt with an unrelated decoy image can sharply lower attack success rate (ASR). The operative change is in the defense pipeline, not in the image. Across five frontier VLMs, two encoded-attack families, and three black-box defenses, a caption-mediated defense (ECSO) that leaves ASR essentially unchanged on text-only encoded input drops it by up to $73$pp once a content-free decoy is attached; every non-saturated contrast is significant under exact McNemar tests. We advance two hypotheses for this pattern, supported by indirect evidence rather than pipeline introspection, since a black-box threat model precludes inspecting vendor internals: caption-mediated defenses branch on image presence, and intrinsic image-side safety engages on image-resident content. Three controls constrain the explanation. Blank-canvas and natural-photograph decoys reproduce the effect on every model, implicating image presence rather than content; the effect replicates on three open-weight VLMs served with no moderation layer, so it is not a vendor-filtering artifact; and a non-symbolic, meaning-based encoder reproduces it, so it is not specific to symbolic obfuscation. Attaching a decoy unconditionally is not deployable --- it raises benign refusal to $20$--$79\%$, an inflation of $+10$ to $+67$pp --- but gating attachment on a lightweight encoded-input detector returns benign refusal to the text baseline while preserving the safety gain wherever the detector fires, making detector recall the binding constraint. Under adaptive attacks that target the caption-mediated re-check, the effect degrades but holds. We frame this as an observation about pipeline interaction, not as a robust defense.
