Search papers, labs, and topics across Lattice.
This paper introduces a systematic evaluation framework to assess the robustness of Vision-Language Models (VLMs) against natural semantic variations across image classification, semantic segmentation, and visual question answering tasks. They evaluate CLIP, robust CLIP, BLIP2, and SigLIP2 on curated adversarial datasets including typographic attacks, ImageNet-A, and natural language-induced adversarial examples. Results show that robust CLIP models can amplify natural adversarial vulnerabilities, and CLIP models exhibit significant performance degradation when faced with natural language-induced adversarial examples.
Robust CLIP models, surprisingly, can be *more* vulnerable to natural semantic variations than standard CLIP, revealing a critical flaw in current robustness training strategies.
Recent advances in vision-language models (VLMs) trained on web-scale image-text pairs have enabled impressive zero-shot transfer across a diverse range of visual tasks. However, comprehensive and independent evaluation beyond standard benchmarks is essential to understand their robustness, limitations, and real-world applicability. This paper presents a systematic evaluation framework for VLMs under natural adversarial scenarios for diverse downstream tasks, which has been overlooked in previous evaluation works. We evaluate a wide range of VLMs (CLIP, robust CLIP, BLIP2, and SigLIP2) on curated adversarial datasets (typographic attacks, ImageNet-A, and natural language-induced adversarial examples). We measure the natural adversarial performance of selected VLMs for zero-shot image classification, semantic segmentation, and visual question answering. Our analysis reveals that robust CLIP models can amplify natural adversarial vulnerabilities, and CLIP models significantly reduce performance for natural language-induced adversarial examples. Additionally, we provide interpretable analyses to identify failure modes. We hope our findings inspire future research in robust and fair multimodal pattern recognition.
