Search papers, labs, and topics across Lattice.
This paper introduces Local Margin Restoration (LMR), a novel one-step Test-Time Adaptation (TTA) framework designed to enhance the robustness of vision-language models (VLMs) like CLIP under distribution shifts. By implementing a Protected Margin Restoration (PMR) objective that preserves local semantic geometry and a dual-stage stabilization mechanism to mitigate bias accumulation, LMR effectively prevents mode collapse during adaptation. Experimental results on CIFAR-C, ImageNet-C, and ImageNet variants reveal that LMR significantly outperforms existing TTA methods, demonstrating its efficiency in low-batch test-time scenarios.
Local Margin Restoration not only shields VLMs from bias but also preserves their semantic integrity, leading to superior performance in dynamic environments.
Vision-language models (VLMs) such as CLIP exhibit remarkable zero-shot capabilities, yet their performance frequently degrades sharply under unexpected test-time distribution shifts. While Test-Time Adaptation (TTA) offers a promising solution, continuously adapting VLMs over an unlabeled test stream presents fundamental challenges. Conventional top-1-centric updates often reinforce errors by corrupting the local semantic geometry among related classes, while iterative adaptation exacerbates progressive bias accumulation, ultimately driving the model toward mode collapse. To overcome these coupled vulnerabilities, we propose Local Margin Restoration (LMR), a lightweight, one-step TTA framework. At the sample level, our Protected Margin Restoration (PMR) objective recovers local semantic geometry by shielding plausible near-top candidates from external hard negatives. Concurrently, to combat stream-level degradation, we introduce a dual-stage stabilization mechanism, featuring an Adaptive Margin (AM) controller and Bias Correction (BC), to dynamically disrupt progressive bias accumulation and prevent mode collapse. Extensive experiments on CIFAR-C, ImageNet-C, and ImageNet variants demonstrate that LMR consistently outperforms state-of-the-art TTA baselines, proving exceptionally robust and efficient even in challenging low-batch test-time regimes. Our code is available at https://github.com/DennisHuangYan/LMR.
