Search papers, labs, and topics across Lattice.
This paper analyzes the InfoNCE contrastive learning objective and demonstrates that it induces Gaussian structure in the learned representations. The authors prove this result in two regimes: first, under alignment and concentration assumptions, and second, by adding a small regularization term promoting low feature norm and high feature entropy. Empirical validation on synthetic and CIFAR-10 datasets confirms the emergence of Gaussian behavior across various encoder architectures, providing a theoretical justification for observed Gaussianity in contrastive representations.
InfoNCE, the workhorse of contrastive learning, secretly forces your representations to be Gaussian.
Contrastive learning has become a cornerstone of modern representation learning, allowing training with massive unlabeled data for both task-specific and general (foundation) models. A prototypical loss in contrastive training is InfoNCE and its variants. In this work, we show that the InfoNCE objective induces Gaussian structure in representations that emerge from contrastive training. We establish this result in two complementary regimes. First, we show that under certain alignment and concentration assumptions, projections of the high-dimensional representation asymptotically approach a multivariate Gaussian distribution. Next, under less strict assumptions, we show that adding a small asymptotically vanishing regularization term that promotes low feature norm and high feature entropy leads to similar asymptotic results. We support our analysis with experiments on synthetic and CIFAR-10 datasets across multiple encoder architectures and sizes, demonstrating consistent Gaussian behavior. This perspective provides a principled explanation for commonly observed Gaussianity in contrastive representations. The resulting Gaussian model enables principled analytical treatment of learned representations and is expected to support a wide range of applications in contrastive learning.
