Search papers, labs, and topics across Lattice.
This paper investigates the vulnerability of multimodal large language models (MLLMs) to adversarial attacks by focusing on their intrinsic attention mechanisms in the frequency domain. The authors discover that MLLMs are particularly sensitive to phase information, which is crucial for interpreting visual structures, and develop a phase-aware adversarial attack framework that targets these phase regions to create effective and imperceptible perturbations. Experimental results show that their approach significantly outperforms existing adversarial methods in misleading MLLMs' attention and predictions.
MLLMs can be misled by focusing attacks on phase information, revealing a critical vulnerability that traditional adversarial methods overlook.
Multimodal large language models (MLLMs) have extended the capability of large language models (LLMs) to process more contextual multimodal information, showing remarkable progress in diverse realistic multimodal applications. Despite their strong perception and reasoning abilities, recent studies reveal that MLLMs remain highly vulnerable to adversarial inputs, especially those targeting visual components. However, existing attacks mainly focus on global perturbations, lacking an understanding of how MLLMs internally interpret visual structures. In this paper, we make the attempt to investigate the intrinsic focus of MLLMs in the frequency domain and discover that their predictions are particularly sensitive to phase information, which encodes essential structural and semantic cues. Based on this observation, we propose a novel phase-aware adversarial attack framework that explicitly restricts adversarial perturbations to structure-relevant phase regions to suppress the MLLMs' focus for effective and imperceptible attacks. To further amplify the structural influence, we also introduce an auxiliary adversarial prompt learning module to guide multimodal misalignment around phase-sensitive regions, misleading the MLLM's attention toward targeted structural patterns. Extensive experiments on multiple representative MLLM models and datasets demonstrate the superior effectiveness of our method compared to existing attacks.
