Search papers, labs, and topics across Lattice.
This paper investigates the vulnerability of Vision-Language Models (VLMs) like CLIP to adversarial attacks and introduces a novel test-time defense mechanism called Directional Bias-guided Defense (DBD). By analyzing the feature space of adversarial images, the authors identify a consistent directional shift that can be leveraged to realign features with their correct class centers. The results show that DBD achieves state-of-the-art adversarial robustness while maintaining clean accuracy, with adversarial accuracy even exceeding clean accuracy in some cases, highlighting the informative nature of adversarial perturbations.
Adversarial images in CLIP reveal a consistent directional bias that can be exploited to enhance model robustness, leading to surprising cases where adversarial accuracy surpasses clean accuracy.
Vision-Language Models (VLMs), such as CLIP, have shown strong zero-shot generalization but remain highly vulnerable to adversarial perturbations, posing serious risks in real-world applications. Test-time defenses for VLMs have recently emerged as a promising and efficient approach to defend against adversarial attacks without requiring costly large-scale retraining. In this work, we uncover a surprising phenomenon: under diverse input transformations, adversarial images in CLIP's feature space consistently shift along a dominant direction, in contrast to the dispersed patterns of clean images. We hypothesize that this dominant shift, termed the Defense Direction, opposes the adversarial shift, pointing features back toward their correct class centers. Building on this insight, we propose Directional Bias-guided Defense (DBD), a test-time framework that estimates the Defense Direction and employs a DB-score-based two-stream reconstruction strategy to recover robust representations. Experiments on 15 datasets demonstrate that DBD not only achieves SOTA adversarial robustness while preserving clean accuracy, but also reveals the counterintuitive result that adversarial accuracy can even surpass clean accuracy. This demonstrates that adversarial perturbations inherently encode directional priors about the true decision boundary.
