Search papers, labs, and topics across Lattice.
This paper introduces LFM, a novel framework that leverages foundation models for source-free universal domain adaptation (SF-UniDA), addressing the limitations of existing methods that rely on inefficient techniques like threshold tuning and clustering. By utilizing a vision-language model to compute similarities between target samples and text labels, including those for unknown classes, the framework effectively identifies label shifts and refines pseudo-labels through a consensus strategy. Extensive experiments across various benchmarks reveal that LFM significantly outperforms traditional SF-UniDA approaches, showcasing its potential for adapting to unlabeled target domains without access to source data.
Leveraging foundation models, LFM achieves superior performance in source-free universal domain adaptation, even in the presence of unknown classes.
Source-free universal domain adaptation (SF-UniDA) adapts a pre-trained source model to an unlabeled target domain under both covariate and label shifts, without access to source data. However, existing SF-UniDA methods rely on inefficient techniques such as threshold tuning and clustering. Foundation models (FMs), known for their generalization and zero-shot capabilities, remain underexplored in SF-UniDA. In this paper, we propose a framework that leverages foundation models (LFM) for SF-UniDA. We use a vision-language model (VLM) to compute similarities between target samples and text labels, including those for unknown classes generated by prompting a large language model. The label shift type is determined by analyzing the coefficient of variation of a similarity-based sample-level score. Unknown samples are identified using a binary Gaussian mixture model fitted to another similarity-based metric. Under a consensus strategy, the pseudo-labels generated by the VLM are refined by the target model initialized with the pre-trained source model, integrating knowledge from both the source domain and foundation models. Finally, these refined pseudo-labels are used to train the target model. Extensive experiments across all possible label shifts and multiple benchmarks demonstrate the effectiveness and superiority of our proposed LFM framework. Our code is available at https://github.com/iamjingli/LFM.
