Search papers, labs, and topics across Lattice.
This paper investigates cross-domain few-shot learning with CLIP and finds that uniformly aligning all image patch tokens with textual embeddings during fine-tuning hurts performance. They discover that actively pushing away low-similarity "tail tokens" from their textual embeddings improves target-domain performance due to overfitting on scarce data. They propose Adaptive Tail-Head Alignment (ATHA), which adaptively aligns tokens based on their semantic content, achieving state-of-the-art results on CDFSL benchmarks.
Counterintuitively, pushing *away* certain image tokens from their corresponding text embeddings boosts CLIP's few-shot cross-domain performance.
Vision-Language Models (VLMs) such as CLIP demonstrate strong zero-shot generalization, but their performance significantly degrades in cross-domain scenarios with scarce target-domain training data (Cross-Domain Few-Shot Learning, CDFSL). In this paper, we focus on the target-domain few-shot finetuning in the CLIP-based CDFSL task. Prevailing finetuning paradigms uniformly align all image patch tokens with their corresponding textual embeddings. However, we find a counterintuitive phenomenon: actively pushing away certain low-similarity image tokens, termed"tail tokens", from their textual embeddings consistently improves target-domain performance. We delve into this phenomenon and provide a novel interpretation: under great domain shifts and scarce training data, the model can hardly extract semantic information from visual inputs; therefore, the common belief of alignment is valid only for tokens already containing sufficient semantic information; for tail tokens, forcing the alignment would lead to excessive overfitting to the scarce training, while breaking the alignment is more useful. Motivated by this, we propose Adaptive Tail-Head Alignment (ATHA), a novel fine-tuning strategy for CLIP that transforms the conventional uniform alignment paradigm to an adaptive alignment paradigm, with both alignment strengthening and weakening. Extensive experiments on four challenging CDFSL benchmarks validate our state-of-the-art performance. Our code is available at https://github.com/shuaiyi308/ATHA.
