Search papers, labs, and topics across Lattice.
The paper identifies high-variance gradients in transfer-based black-box attacks on LVLMs, specifically in M-Attack, due to ViT translation sensitivity and structural asymmetries between source and target crops. To address this, they introduce M-Attack-V2, which incorporates Multi-Crop Alignment (MCA) for source-side gradient averaging and Auxiliary Target Alignment (ATA) to create a smoother target manifold. M-Attack-V2 significantly improves attack success rates on state-of-the-art LVLMs like Claude-4.0, Gemini-2.5-Pro, and GPT-5, demonstrating the effectiveness of their gradient-denoising approach.
LVLMs aren't as robust as you think: a simple tweak to existing black-box attacks can triple the success rate against Claude-4.0 and achieve near-perfect scores against Gemini 2.5 Pro and GPT-5.
Black-box adversarial attacks on Large Vision-Language Models (LVLMs) are challenging due to missing gradients and complex multimodal boundaries. While prior state-of-the-art transfer-based approaches like M-Attack perform well using local crop-level matching between source and target images, we find this induces high-variance, nearly orthogonal gradients across iterations, violating coherent local alignment and destabilizing optimization. We attribute this to (i) ViT translation sensitivity that yields spike-like gradients and (ii) structural asymmetry between source and target crops. We reformulate local matching as an asymmetric expectation over source transformations and target semantics, and build a gradient-denoising upgrade to M-Attack. On the source side, Multi-Crop Alignment (MCA) averages gradients from multiple independently sampled local views per iteration to reduce variance. On the target side, Auxiliary Target Alignment (ATA) replaces aggressive target augmentation with a small auxiliary set from a semantically correlated distribution, producing a smoother, lower-variance target manifold. We further reinterpret momentum as Patch Momentum, replaying historical crop gradients; combined with a refined patch-size ensemble (PE+), this strengthens transferable directions. Together these modules form M-Attack-V2, a simple, modular enhancement over M-Attack that substantially improves transfer-based black-box attacks on frontier LVLMs: boosting success rates on Claude-4.0 from 8% to 30%, Gemini-2.5-Pro from 83% to 97%, and GPT-5 from 98% to 100%, outperforming prior black-box LVLM attacks. Code and data are publicly available at: https://github.com/vila-lab/M-Attack-V2.
