Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
ViTAMINS integrates synthetic hard negatives into the pretraining of vision transformers to enhance representation quality in a self-supervised manner. This method was rigorously evaluated across various tasks, including image retrieval and segmentation, revealing that the learned representations not only capture semantic content but also act as effective classifiers, achieving up to an 11.3% improvement over existing baselines. The findings suggest that contrastive learning can be a more efficient and powerful alternative to current generative and self-distillation methods in vision tasks.
Synthetic hard negatives can boost vision transformer performance by over 11%, challenging the dominance of generative approaches in self-supervised learning.
We introduce ViTAMINS, a method that integrates synthetic hard negatives into unsupervised vision transformer pretraining to improve representation quality. Our approach is thoroughly benchmarked on ImageNet and transfer learning, image retrieval, copy detection, and image, video segmentation tasks. Notably, our proposed negatives give rise to emergent properties, where learned representations contain explicit information about the semantic content of an image and serve as excellent classifiers (up to +11.3% over baselines). ViTAMINS achieves these benefits through simple modifications to existing contrastive frameworks and outperforms competing methods while being more resource efficient, e.g., our ViT-B surpasses V-JEPA with ViT-L. Our findings motivate reconsidering contrastive learning as a simpler yet powerful alternative to dominant generative and self-distillation approaches.