Search papers, labs, and topics across Lattice.
This paper addresses the problem of unreliable view selection in Test-Time Prompt Tuning (TPT) by introducing a dual-modality anchor-guided filtering framework. They use text anchors from attribute-rich descriptions and adaptive image anchors to filter augmented views based on alignment and confidence with these anchors. By ensembling predictions from the anchors with the original model output, they achieve more stable supervision for prompt updates, leading to state-of-the-art results across 15 datasets.
No more garbage in, garbage out: Anchor-guided filtering finally makes test-time prompt tuning reliable by grounding view selection in semantic evidence.
Test-Time Prompt Tuning (TPT) adapts vision-language models using augmented views, but its effectiveness is hindered by the challenge of determining which views are beneficial. Standard entropy-based filtering relies on the internal confidence scores of the model, which are often miscalibrated under distribution shift, assigning high confidence to irrelevant crops or background regions while ignoring semantic content. To address this, we propose a dual-modality anchor-guided framework that grounds view selection in semantic evidence. We introduce a text anchor from attribute-rich descriptions, to provide fine-grained class semantics, and an adaptive image anchor that captures evolving test-time statistics. Using these anchors, we filter views based on alignment and confidence, ensuring that only informative views guide adaptation. Moreover, we treat the anchors as auxiliary predictive heads and combine their predictions with the original output in a confidence-weighted ensemble, yielding a stable supervision signal for prompt updates. Extensive experiments on 15 benchmark datasets demonstrate new state-of-the-art performance, highlighting the contribution of anchor-guided supervision as a foundation for robust prompt updates.
