Search papers, labs, and topics across Lattice.
This paper introduces InfraPatch, a targeted attack framework specifically designed for infrared vision-language models (IR-VLMs), which optimizes grayscale patches to induce desired semantic behaviors across multiple tasks. By evaluating ten IR-VLM variants on synthetic infrared images, the authors demonstrate that InfraPatch achieves targeted attack success rates ranging from 86% to 100%, highlighting significant vulnerabilities in these models. The findings underscore the necessity for enhanced robustness assessments in infrared multimodal systems, given the effectiveness of localized perturbations in manipulating model outputs.
Small grayscale patches can exploit vulnerabilities in infrared vision-language models, achieving up to 100% success in targeted attacks across diverse tasks.
Infrared vision-language models (IR-VLMs) have emerged as a promising paradigm for multimodal perception under low-visibility conditions, yet their robustness to targeted adversarial attacks remains poorly understood. Existing adversarial patch methods mainly study RGB-based models or a single downstream task and do not characterize whether localized perturbations can induce an intended semantic target in IR-VLMs. We propose InfraPatch, a white-box, per-instance framework for targeted digital grayscale patch attacks against IR-VLMs. InfraPatch optimizes a compact single-channel patch within an approximately 5% local-area budget, combines proxy-guided placement with task-adaptive semantic objectives, and induces target behaviors in image classification, image captioning, and binary visual question answering. We evaluate ten infrared-adapted model variants on 300 synthetic infrared-style images generated by applying DiffV2IR to a fixed 30-category COCO subset, using clean-conditioned targeted success criteria. InfraPatch achieves targeted attack success rates from 86.00% to 100% across the ten variants. On CLIP and BLIP-2, proxy location search improves success by 6.67 and 10.33 percentage points over optimized random placement, respectively; LLaVA-1.5 remains saturated near 100% under both settings. Patch-area and objective ablations further expose substantial differences in vulnerability across architectures and task formats. These results show that small grayscale patches can inject chosen target semantics across IR-VLM families under a controlled digital threat model, motivating stronger robustness evaluation for infrared multimodal systems.
