Search papers, labs, and topics across Lattice.
This paper introduces a hierarchical adversarial fine-tuning framework for VLMs to improve robustness against adversarial attacks, particularly those targeting superclasses. The method aligns image-text modalities at multiple levels of a class hierarchy using hierarchical embeddings and margin-based losses. Results demonstrate improved adversarial robustness and natural performance compared to standard fine-tuning approaches, especially when aligning over multiple class hierarchies.
Adversarial training can be made more effective by considering the hierarchical relationships between classes, leading to vision-language models that are more robust to attacks on both specific classes and their broader categories.
Vision-Language Models (VLMs) can perform zero-shot classification but are susceptible to adversarial attacks. While robust fine-tuning improves their robustness, existing approaches align fixed text embeddings with an image embedding, sacrificing natural performance and robustness. A robustness degradation also occurs when a model faces adversarial attacks targeting superclasses (parent classes, e.g., mammal) in addition to their base (leaf) classes (e.g., cat). Thus, to enhance adversarial robustness and leverage the inherent hierarchical properties of class space, we propose a novel adversarial fine-tuning framework based on hierarchical embeddings and several levels of adversarially robust alignment of image-text modalities. Additional mechanisms place visual embeddings at the desired depth of hierarchy, and we provide a theoretical connection between the depth of embedding in the hierarchy and the maximum viable margin size. Our model naturally realizes several margin sizes, boosting generalization of adversaries for robustification. As various trees with different parent labels can share the same leaf labels, we also consider aligning over multiple trees to boost semantic variety. Experiments across several datasets are performed.
