Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Test-Time Logit Prompting (TLP), a novel framework designed for adapting vision-language models (VLMs) to scenarios where input modalities are missing, without relying on source training data. By employing uncertainty-aware adjustments and modality-complete consistency regularization, TLP effectively mitigates prediction shifts caused by missing information, enhancing model performance. Experimental results across various benchmarks reveal that TLP can improve recognition accuracy by up to 8% with minimal tunable parameters and few optimization steps at test time.
TLP enables vision-language models to adapt on-the-fly to missing modalities, achieving significant performance gains without needing access to original training data.
Vision-language models (VLMs) have achieved remarkable performance by leveraging complementary information from large-scale image-text pairs. However, missing-modality inputs are commonly encountered during real-world deployment, often leading to significant performance degradation. Existing methods primarily enhance model robustness by learning modality compensation strategies from source training data. However, their reliance on source training data makes them difficult to apply when original data are unavailable due to privacy, storage, or accessibility constraints, such as clinical applications and personalized AI services. This raises an important yet underexplored question: can VLMs be efficiently adapted at test time for visual recognition with missing modalities without accessing source training data? To this end, we propose Test-Time Logit Prompting (TLP), a lightweight source-free test-time adaptation framework for visual recognition with missing modalities. To address missing-induced prediction shifts, TLP optimizes logit prompts with uncertainty-aware adjustment and modality-complete consistency regularization, adaptively adjusting prediction confidence while preserving semantic consistency. Extensive experiments across diverse vision-language benchmarks demonstrate that TLP consistently enhances recognition performance under missing-modality scenarios, achieving up to 8\% improvements while requiring only hundreds of tunable parameters and a few test-time optimization steps.