Search papers, labs, and topics across Lattice.
This paper introduces Consensus-Driven Shift Modulation (COSMO), a novel approach for source-free domain adaptation (SFDA) that addresses the challenge of source-derived evidence forgetting by leveraging pretrained vision-language models (VLMs). By formulating the adaptation process as a sample-wise reliability-allocation problem, COSMO creates a dynamic consensus that balances the contributions of the source model and VLM based on their reliability across target samples. Experimental results demonstrate that COSMO achieves state-of-the-art performance on four benchmarks while effectively retaining valid source-derived evidence and integrating complementary VLM insights.
COSMO redefines source-free domain adaptation by dynamically balancing the influence of source models and vision-language models, achieving unprecedented performance while mitigating source bias.
Source-free domain adaptation (SFDA) adapts a source-trained model to an unlabeled target domain without source data, a practical setting under privacy or storage constraints. Yet its self-generated supervision can reinforce source bias under substantial domain shifts. Pretrained vision-language models (VLMs) offer complementary semantic knowledge, but the relative reliability of the source model and VLM varies across target samples. Existing cross-model guidance does not explicitly account for this variation and may overwrite valid source-derived evidence under conflict, a failure we term source-derived evidence forgetting. We formulate VLM-guided SFDA as a sample-wise reliability-allocation problem and propose Consensus-Driven Shift Modulation (COSMO). COSMO replaces expert-to-expert guidance with co-adaptation through an anchored shared consensus. It first forms a sample-specific initial consensus that favors the more concentrated prediction. During adaptation, COSMO re-aggregates both branches'evolving evidence and regulates how far the resulting consensus moves from its initial anchor based on consensus uncertainty and training progress. This keeps the shared supervision anchored yet adaptive. Across four benchmarks, COSMO achieves state-of-the-art performance under matched VLM backbones. Further analyses indicate that it better balances the retention of valid source-derived evidence with the absorption of complementary VLM evidence.
