Search papers, labs, and topics across Lattice.
This paper introduces an evolutionary-computation-guided framework for generating cross-modal adversarial attacks on vision-language models (VLMs), addressing their vulnerabilities to adversarial perturbations. By simultaneously evolving hard negative semantic embeddings in the textual domain and maintaining a population of object-region perturbations in the visual domain, the method enhances the exploration of attack trajectories across multiple tasks. Experimental results demonstrate that this co-evolutionary approach significantly improves attack performance on various vision-language tasks while ensuring the feasibility of perturbations and maintaining high fitness levels.
Evolving adversarial attacks across both text and image modalities can dramatically enhance the transferability and effectiveness of adversarial perturbations in vision-language models.
Vision-language models (VLMs) exhibit strong generalization across multimodal tasks but remain vulnerable to adversarial perturbations. Existing attacks typically follow single-trajectory gradient optimization or task-specific objectives, limiting search-space exploration and cross-task transferability. We propose an evolutionary-computation-guided cross-modal attack framework for unified VLMs. The framework adaptively searches both textual and visual spaces. On the textual side, it evolves hard negative semantic embeddings around the source-category representation to provide diverse cross-modal repulsion. On the visual side, it maintains a population of object-region perturbations and combines momentum-based gradient updates with evolutionary selection, mutation, and crossover to more reliably explore multiple feasible trajectories. Jointly optimizing semantic negative guidance and localized perturbations generates adversarial examples that consistently shift source-object semantics toward target categories across vision-language tasks. Theoretical analyses show that the co-evolutionary search preserves perturbation feasibility, prevents degradation of the best observed fitness, and increases the probability of reaching high-margin adversarial regions compared with single-trajectory optimization. Experiments on Florence-2, OFA, and UnifiedIO-2 demonstrate strong overall attack performance across image captioning, object detection, region categorization, and object localization. Ablation studies further verify the complementary effectiveness of text-side semantic evolution and image-side perturbation evolution, as well as the framework's efficiency and cross-task transferability.
