Search papers, labs, and topics across Lattice.
This paper introduces QuISE, a novel defense mechanism against typographic attacks on vision-language models (VLMs) that operates without requiring model-specific modifications or additional training. By employing influence-aware text localization, QuISE identifies and replaces vulnerable text regions with semantically irrelevant content, ensuring that the model's responses are based on visual evidence rather than adversarial text. Experimental results demonstrate that QuISE significantly enhances defended accuracy across multiple benchmarks and VLMs, achieving a recovery rate of 67.9-75.0% while maintaining a low harm rate of 0.5-1.1%.
QuISE effectively neutralizes typographic attacks on VLMs without any model-specific adjustments, achieving impressive accuracy recovery rates.
Typographic attacks pose a critical threat to vision-language models (VLMs) by injecting misleading text into images and causing models to rely on adversarial textual cues rather than visual evidence. Existing defenses often require model-specific modifications, additional training, or access to internal model components, limiting their applicability to modern closed-source VLMs. In this paper, we propose QuISE, a model-agnostic, training-free black-box defense based on query-irrelevant semantic editing. QuISE first identifies text regions likely to affect the current query through influence-aware text localization. QuISE then replaces these regions with two semantically distinct replacement texts that are irrelevant to both the query and the image. The final answer is determined by answer consistency across the edited images. Extensive experiments on three typographic-attack benchmarks, four attack settings, and four VLMs show that QuISE consistently improves defended accuracy. QuISE achieves a recovery rate of 67.9-75.0% with a harm rate of 0.5-1.1%.
