Search papers, labs, and topics across Lattice.
This paper addresses the challenge of distribution shifts in multi-modal test-time adaptation (TTA) by explicitly modeling category-conditional distributions. They introduce a probabilistic Gaussian model tailored for multi-modal TTA and an adaptive contrastive asymmetry rectification technique to mitigate the impact of modality asymmetry. Experiments on various benchmarks demonstrate state-of-the-art performance under distribution shifts, indicating improved prediction accuracy and decision boundary reliability.
Multi-modal models can now better handle distribution shifts thanks to a new method that explicitly models how different categories are distributed, even when the modalities are asymmetrical.
Multi-modal test-time adaptation (TTA) enhances the resilience of benchmark multi-modal models against distribution shifts by leveraging the unlabeled target data during inference. Despite the documented success, the advancement of multi-modal TTA methodologies has been impeded by a persistent limitation, i.e., the lack of explicit modeling of category-conditional distributions, which is crucial for yielding accurate predictions and reliable decision boundaries. Canonical Gaussian discriminant analysis (GDA) provides a vanilla modeling of category-conditional distributions and achieves moderate advancement in uni-modal contexts. However, in multi-modal TTA scenario, the inherent modality distribution asymmetry undermines the effectiveness of modeling the category-conditional distribution via the canonical GDA. To this end, we introduce a tailored probabilistic Gaussian model for multi-modal TTA to explicitly model the category-conditional distributions, and further propose an adaptive contrastive asymmetry rectification technique to counteract the adverse effects arising from modality asymmetry, thereby deriving calibrated predictions and reliable decision boundaries. Extensive experiments across diverse benchmarks demonstrate that our method achieves state-of-the-art performance under a wide range of distribution shifts. The code is available at https://github.com/XuJinglinn/AdaPGC.
