Search papers, labs, and topics across Lattice.
This paper investigates the attention sink phenomenon in Vision-Language Models (VLMs) within the context of Cross-Domain Few-Shot Learning (CDFSL), revealing that standard fine-tuning exacerbates the issue, leading to poor class discriminability. The authors attribute this to the model's shortcut learning for domain adaptation, where it overly relies on "simple" tokens close to target classes. To mitigate this, they propose a dynamic token re-weighting strategy during fine-tuning that suppresses reliance on simple tokens and enhances learning of "hard" tokens, achieving state-of-the-art results on four benchmark datasets.
Fine-tuning vision-language models for cross-domain few-shot learning makes them *worse* at distinguishing between classes due to an exacerbated "attention sink" problem, but a simple token re-weighting scheme can fix it.
Vision-language models (VLMs) like CLIP have shown impressive generalization capabilities, yet their potential for Cross-Domain Few-Shot Learning (CDFSL) remains underexplored, where the model needs to transfer source-domain information to target domains with scarce training data. While the attention sink phenomenon has been observed in VLMs for certain tasks, its role in CDFSL scenarios has not been studied. In this paper, we uncover a critical issue overlooked by prior works: standard target-domain few-shot fine-tuning in CDFSL significantly exacerbates the attention sink problem, leading to poor discriminability across classes. To understand this phenomenon, through extensive experiments, we interpret it as the model's shortcut learning for domain adaptation: to overcome the huge domain gap between the source and target domains, the model shows a high tendency to push tokens that are initially closer to target-domain classes (i.e., simple tokens) to be even closer to these classes, exacerbating the attention sink and wasting the capability of learning other discriminative but initially further tokens (i.e., hard tokens). To address this, we propose a novel approach to dynamically re-weight tokens according to their relevance with target-domain classes during the target-domain finetuning, which explicitly suppresses the model's reliance on these simple tokens and enhances the learning of hard tokens, reducing sink tokens and enhancing discriminability. Extensive experiments on four benchmark datasets validate the rationale of our method, demonstrating new state-of-the-art performance. Our codes are available at https://github.com/shuaiyi308/TIR.
