Search papers, labs, and topics across Lattice.
The paper introduces Paraphrase-Decomposition-Aggregation (PDA), a training-free defense framework that enhances the robustness of Vision-Language Models (VLMs) against adversarial image attacks by leveraging text augmentation. PDA operates at test time through prompt paraphrasing, question decomposition, and consistency aggregation, requiring no model retraining. Experiments across VQA, classification, and captioning tasks demonstrate that PDA provides consistent robustness gains against various adversarial perturbations while maintaining competitive clean accuracy.
VLMs can be made robust to adversarial image attacks without retraining, using a clever text-augmentation scheme at inference time.
Vision-language models (VLMs) are vulnerable to adversarial image perturbations. Existing works based on adversarial training against task-specific adversarial examples are computationally expensive and often fail to generalize to unseen attack types. To address these limitations, we introduce Paraphrase-Decomposition-Aggregation (PDA), a training-free defense framework that leverages text augmentation to enhance VLM robustness under diverse adversarial image attacks. PDA performs prompt paraphrasing, question decomposition, and consistency aggregation entirely at test time, thus requiring no modification on the underlying models. To balance robustness and efficiency, we instantiate PDA as invariants that reduce the inference cost while retaining most of its robustness gains. Experiments on multiple VLM architectures and benchmarks for visual question answering, classification, and captioning show that PDA achieves consistent robustness gains against various adversarial perturbations while maintaining competitive clean accuracy, establishing a generic, strong and practical defense framework for VLMs during inference.
