Search papers, labs, and topics across Lattice.
This paper introduces CARE, a collaborative adversarial fine-tuning framework that integrates multiple robust model experts in vision-language models (VLMs) to enhance adversarial robustness. By facilitating knowledge exchange through embedding-space harmonization, CARE allows these specialized models to retain their unique strengths while benefiting from collective learning. Experimental results demonstrate that CARE significantly outperforms individually fine-tuned models across various tasks, highlighting the effectiveness of collaborative learning in improving robustness against adversarial attacks.
Collaborative learning among specialized model experts can dramatically enhance adversarial robustness in vision-language models, outperforming traditional fine-tuning methods.
Vision-language models (VLMs), such as CLIP, are vulnerable to adversarial attacks, posing a serious problem for real-life applications and deployment. Adversarial fine-tuning emerges as a prominent defense method; however, different fine-tuning strategies often produce specialized models with distinct robustness characteristics. Each fine-tuned model in turn thrives in some evaluation settings but falters on others, limiting their defensive capabilities. We refer to these specialized fine-tuned models as robust model experts and propose a collaborative adversarial fine-tuning framework: CARE - Collaborative Adversarial Robustness fine-tuning using Embedding alignment. CARE maintains multiple experts during training, enables knowledge exchange through embedding-space harmonization, and consolidates the learned knowledge into a single unified robust model. Experts benefit from one another while preserving their individual specializations, enabling the final model to inherit complementary robustness properties. In this paper, we demonstrate CARE on two different adversarial fine-tuning strategies with complementary robustness behaviors. Extensive experiments on classic image classification and downstream vision-language tasks display the effectiveness of our approach, with CARE being able to outperform individually learned model experts. The results suggest that collaborative learning across model experts is a promising direction for improving adversarial robustness.
