Search papers, labs, and topics across Lattice.
This paper introduces a novel probabilistic model for the CLIP latent space, leveraging Mixtures of von Mises-Fisher (MovMF) distributions to better capture its intrinsic hyperspherical geometry. By employing the Expectation-Maximization (EM) algorithm, the authors demonstrate that their model enhances long-tailed and out-of-distribution detection while providing a clear semantic decomposition of embeddings. The findings indicate that CLIP representations are more accurately described as a hyperspherical semantic mixture, challenging the conventional Gaussian assumptions in existing models.
CLIP's latent space is better understood as a hyperspherical semantic mixture, revealing richer and more interpretable representations than traditional Gaussian models.
Contrastive Language-Image Pretraining (CLIP) representations form a semantic embedding space governed by cosine similarity, reflecting an intrinsic hyperspherical geometry. However, existing probabilistic interpretations typically rely on Gaussian assumptions, which fail to capture this directional and multimodal structure. We propose a principled density model for the CLIP latent space based on Mixtures of von Mises-Fisher (MovMF) distributions defined on the unit hypersphere. Using the Expectation-Maximization (EM) algorithm, we efficiently learn a probabilistic model in which each mixture component corresponds to a coherent semantic concept. This formulation yields a closed-form likelihood naturally aligned with hyperspherical geometry, enabling accurate and interpretable density estimation. Empirically, our model significantly improves long-tailed and out-of-distribution detection and provides a natural semantic decomposition, representing each embedding as a sparse probabilistic combination of interpretable concepts. These results suggest that CLIP latent space is more faithfully characterized as a hyperspherical semantic mixture rather than an isotropic Gaussian, establishing a simple and geometrically consistent probabilistic framework for modeling and understanding multimodal representations. Project page is available at https://xiaoyuzhizi.github.io/movmf-clip/.
