Search papers, labs, and topics across Lattice.
This paper introduces a unified approach to improve vision-language models (VLMs) by aligning their inference and adaptation processes through optimal transport (OT) principles. By formulating zero-shot image classification as a cross-modal alignment problem using a Wasserstein OT framework, the authors generate robust pseudo-labels that enhance adaptation without noise amplification. The proposed method, \algname, leverages a soft-label InfoNCE loss to explicitly model relationships between image-text pairs, resulting in performance improvements of up to 7% over existing methods while maintaining efficiency.
Bridging the gap between inference and adaptation in VLMs could lead to significant performance boosts by ensuring robust pseudo-labels that accurately reflect sample-level relationships.
Vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities yet remain sensitive to real-world distribution shifts during inference. Although significant efforts are devoted to adapting VLMs at test time, they rely heavily on noisy pseudo-labels predicted directly from raw embedding similarities during inference, which are unreliable under distribution shift and mislead the adaptation. To avoid noise amplification, existing works craft coarse-grained surrogate objectives during adaptation, which fail to explicitly model sample-level relationships across different modalities, creating objective mismatch with inference, thus leading to marginal performance improvement. In this work, we aim to bridge the detached objectives of inference and adaptation for VLMs, and propose a principled VLM TTA method called \algname. For VLM inference, we formulate the zero-shot image classification task as a cross-modal alignment problem encoded via a Wasserstein OT formulation, providing robust pseudo-labels at the sample-level to effectively adapt VLMs. For VLM adaptation, we adopt a soft-label InfoNCE loss to adapt VLMs based on the OT-induced pseudo-labels, leveraging fine-grained supervisions to explicitly model relationships of individual image-text pairs via contrastive learning, which empowers accurate inference at the same granularity. Moreover, we theoretically reveal that the InfoNCE loss can be neatly reformulated as a Wasserstein OT formulation, thereby unifying the objectives of the inference and adaptation of VLMs to achieve their mutual benefits. Extensive experiments demonstrate the effectiveness and efficiency of our methods, outperforming the best-performing methods by up to 7% with state-of-the-art efficiency.
