Search papers, labs, and topics across Lattice.
This paper introduces Test-time Visual Subspace Steering (T-VSS), a novel approach that enhances the adversarial robustness of vision-language models (VLMs) by adapting directly in the visual feature space during test time. Unlike previous methods that rely on indirect optimization paths, T-VSS constructs a sample-specific low-rank subspace from multi-view feature residuals, allowing for more stable and discriminative predictions. Experimental results demonstrate that T-VSS significantly improves robustness against adversarial attacks while preserving competitive clean accuracy and offering better efficiency compared to existing test-time adaptation techniques.
Adapting visual features directly in a low-rank subspace can drastically improve VLMs' adversarial robustness without sacrificing performance.
Vision-language models (VLMs) achieve strong zero-shot recognition, but they remain highly vulnerable to adversarial perturbations. Recent test-time adaptations improve robustness without retraining, but they do not directly adapt the corrupted visual representation itself. Prompt-based methods adapt the learnable text prompts, while input-space methods optimize pixels or padding at test time. These approaches can improve predictions, but they do so through an indirect and expensive optimization path. We propose Test-time Visual Subspace Steering (T-VSS), a lightweight defense that performs test-time adaptation directly in the visual feature space. T-VSS first builds a sample-specific low-rank subspace from multi-view feature residuals anchored at the attacked image. It then learns a shared feature correction within this subspace using reliability-weighted entropy minimization. By constraining adaptation to a compact visual geometry, T-VSS steers attacked features toward more stable and discriminative predictions while avoiding noisy full-space updates. Experiments on fine-grained, ImageNet, and ImageNet-OOD benchmarks show that T-VSS improves adversarial robustness while maintaining competitive clean accuracy and better efficiency than prior test-time adaptations.
