Search papers, labs, and topics across Lattice.
To resolve how distributional geometry versus pairwise alignment impacts VLM retrieval, this work formulates UOT-Gap, a training-free diagnostic applying unbalanced entropic optimal transport to frozen image-text representations. Benchmarked across CLIP, OpenCLIP, and SigLIP on Flickr8K and COCO-1K, the framework's pair-aware residual tracks retrieval degradation under caption corruptions with a mean absolute Spearman correlation of 0.973, outperforming the standard mean gap baseline of 0.392. Crucially, enforcing geometric gap reduction via UOT barycentric updates actively degrades retrieval performance, proving that naive geometric transport descent decouples from downstream task quality.
Shrinking the geometric modality gap in VLMs can paradoxically destroy retrieval performance, exposing a critical disconnect between representation geometry and task alignment that unbalanced optimal transport tracks with near-perfect fidelity (0.973 vs. 0.392 Spearman).
Vision-language models such as CLIP embed images and text in a shared space, where modality-specific distributions often remain separated. Existing accounts connect this modality gap to initialization, contrastive dynamics, and information imbalance, while its distributional and pairwise contributions to retrieval remain unresolved. We introduce UOT-Gap, a training-free variational diagnostic that models frozen image and text embeddings with unbalanced entropic optimal transport (UOT). The UOT optimum separates transport, coupling complexity, and marginal mass variation; a complementary pair-aware residual compares observed image-caption pairs with the UOT soft matching. On Flickr8K and COCO-1K with frozen CLIP, OpenCLIP, and SigLIP encoders, caption degradation reduces Flickr8K Recall@1 from 0.559 to 0.003. Across six dataset-model conditions, the pair-aware residual tracks retrieval degradation with mean absolute Spearman 0.973, compared with 0.392 for the mean gap. The association remains stable across five random COCO-1K subsets at $0.954\pm0.026$, with a minimum of 0.943. UOT barycentric updates reduce the transport objective while degrading retrieval, distinguishing geometric objective descent from task improvement. These results establish UOT-Gap as a diagnostic for caption quality, modality alignment, and retrieval robustness.
