Search papers, labs, and topics across Lattice.
This paper introduces Structure-Induced Guided Neutralization (SIGN), a plug-and-play defense against adversarial perturbations in LVLMs that leverages prior structural extraction to improve cross-modal alignment. SIGN uses dynamic guided neutralization to efficiently suppress perturbations with minimal image modification (0.5% pixel change). Experiments demonstrate an 87% defense success rate with low computational overhead (0.16 seconds per image) while preserving visual representation quality and benign task performance.
Achieve robust LVLM defense against adversarial attacks with minimal image modification and computational cost by exploiting structural priors.
Image inputs enable Large Vision Language Models (LVLMs) to perceive fine-grained visual information, but also introduce a pixel-level attack surface through which adversarial perturbations can elicit unsafe model behaviors. However, most existing defenses are designed for traditional computer vision settings and thus often overlook the cross-modal alignment required by LVLMs, leading to degraded performance. Meanwhile, the limited defenses tailored to LVLMs often require substantial image modifications and introduce considerable computational overhead, thereby compromising inference quality and efficiency. To address these limitations, we propose Structure-Induced Guided Neutralization (SIGN), a lightweight, plug-and-play defense framework that improves LVLM compatibility via Prior Structural Extraction and achieves efficient perturbation suppression via Dynamic Guided Neutralization. Extensive experiments show that SIGN achieves over 87\% defense success rate with only 0.5\% pixel modification and 0.16 seconds per image, while nearly preserving original visual representations and benign task performance. Our work offers a lightweight alternative to defenses that require costly model training and highlights the potential of exploiting a vision encoder for efficient adversarial protection. Our code is open source on https://anonymous.4open.science/r/SIGN-BCB1.
