Search papers, labs, and topics across Lattice.
This paper introduces a novel defense framework against backdoor attacks in multimodal LLMs by leveraging patch augmentation and cross-view regularization. The method exploits the abnormal invariance of backdoor responses to non-semantic perturbations by using patch-level data augmentation combined with cross-view output difference regularization. By also imposing output entropy constraints, the framework effectively reduces attack success rates while preserving normal text generation capabilities.
Multimodal LLMs are surprisingly vulnerable to backdoor attacks, but a simple patch-based augmentation and cross-view regularization can drastically improve robustness without sacrificing performance.
Multimodal large language models have become an important infrastructure for unified processing of visual and linguistic tasks. However, such models are highly susceptible to backdoor implantation during supervised fine-tuning and will steadily output the attacker's predefined harmful responses once a specific trigger pattern is activated. The core challenge of backdoor defense lies in suppressing attack success under low poisoning ratios while preserving the model's normal generation ability. These two objectives are inherently conflicting. Strong suppression often degrades benign performance, whereas weak regularization fails to mitigate backdoor behaviors. To this end, we propose a unified defense framework based on patch augmentation and cross-view regularity, which simultaneously constrains the model's anomalous behaviors in response to triggered patterns from both the feature representation and output distribution levels. Specifically, patch-level data augmentation is combined with cross-view output difference regularization to exploit the fact that backdoor responses are abnormally invariant to non-semantic perturbations and to proactively pull apart the output distributions of the original and perturbed views, thereby significantly suppressing the success rate of backdoor triggering. At the same time, we avoid over-suppression of the model during defense by imposing output entropy constraints, ensuring the quality of normal command generation. Experimental results across three models, two tasks, and six attacks show that our proposed defense method effectively reduces the attack success rate while maintaining a high level of normal text generation capability. Our work enables the secure, controlled deployment of large-scale multimodal models in realistic low-frequency poisoning and covert triggering scenarios.
