Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces AspectCLIP, a novel framework that optimizes the representation space of Contrastive Language-Image Pretraining (CLIP) by addressing the inherent information asymmetry between images and their textual descriptions. By partitioning training samples into attribute clusters based on textual similarity, AspectCLIP applies cyclic consistency within coherent groups while limiting cross-cluster regularization to prototype-level comparisons. The results show that this aspect-guided approach significantly enhances performance on downstream tasks and leads to a more structured representation space compared to traditional methods.
AspectCLIP reveals that respecting the one-to-many relationship between images and text can dramatically improve representation quality in CLIP models.
Contrastive Language-Image Pretraining learns a shared representation space through large-scale contrastive learning. However, existing methods that enforce global consistency regularization overlook a key challenge: the inherent information asymmetry between images and text: captions typically describe only one specific aspect of an image, thus images with similar visual content can be paired with completely divergent textual content and semantic information. Consequently, global regularizers inadvertently impose constraints between visually similar images whose captions describe divergent aspects, introducing semantic distortion into the representation space. We propose AspectCLIP, a framework that reformulates consistency regularization to respect this one-to-many structure. AspectCLIP first partitions training samples into attribute clusters based on textual similarity to identify aspect-coherent groups, then applies full cyclic consistency within each cluster while restricting cross-cluster regularization to prototype-level comparisons. This aspect-guided regularization enforces strict geometric alignment only when images and texts describe a consistent facet, while allowing flexibility across divergent aspects. Extensive experiments on downstream tasks demonstrate that AspectCLIP consistently outperforms traditional methods and achieves a more structured representation space.