Search papers, labs, and topics across Lattice.
This paper investigates the adversarial vulnerability of transformer-based vision-language models (VLMs) by analyzing the spectral structure of their intermediate linear transformations. The authors introduce a novel white-box attack method, SSGRA, which leverages spectral decompositions to enhance the effectiveness of adversarial attacks compared to existing techniques. Key findings reveal that understanding the spectral properties of these models not only improves attack success rates but also offers new insights into their robustness, highlighting a critical area for future research.
Spectral analysis reveals that transformer-based vision-language models are more vulnerable to adversarial attacks than previously understood, with a new attack method significantly boosting effectiveness.
Adversarial vulnerability in deep neural networks (DNNs) has been studied from the perspectives of decision-boundary geometry, feature robustness, input-output Jacobians, and the instability of inverse problems. Here, we focus on the spectral structure of intermediate linear transformations that propagate information through modern DNNs, an unexplored mechanism of adversarial vulnerability. Specifically, we investigate transformer-based vision-language models, whose linear layers admit interpretable spectral decompositions and whose widespread adoption makes understanding their robustness increasingly important. We propose a white-box spectral-subspace-guided attack (SSGRA) that aligns intermediate representations with the subspace spanned by the bottom right singular vectors. Our experiments show improved attack effectiveness over existing baselines. In addition, SSGRA offers a spectral interpretation of adversarial vulnerability in VLMs, providing insights for improving their robustness.
