Search papers, labs, and topics across Lattice.
This study investigates the performance of various self-supervised learning (SSL) objectives for image and video pretraining under constrained data and computational resources. By comparing contrastive, reconstruction, feature-prediction, and diffusion methods, the research identifies that DINOv2-style pretraining achieves superior performance, particularly when combined with video SSL objectives like VideoMAE, although this combination introduces trade-offs in video tracking and camera-pose estimation. The findings underscore the potential for effective visual model training with limited resources while emphasizing the need for better methods to balance different types of representation learning.
DINOv2-style pretraining outperforms other SSL methods in resource-limited settings, but combining it with video objectives reveals critical trade-offs in performance.
Visual foundation models are a cornerstone of image and video understanding but typically require large amounts of data and computation. The current scale required for pretraining visual foundation models may be unsustainable or unnecessary, and significant benefits arise when effective models can be obtained with fewer resources. To better understand how self-supervised learning (SSL) objectives behave under resource constraints, we conduct a controlled study of image and video SSL objectives under matched data, architecture, and compute budgets. We compare contrastive, reconstruction, feature-prediction, and diffusion objectives and evaluate both standalone and jointly trained image-video SSL formulations across a diverse set of image and video understanding tasks. Our results show that DINOv2-style pretraining consistently provides the strongest overall performance under limited resources. Furthermore, combining DINOv2 with video SSL objectives such as VideoMAE substantially improves image classification and segmentation performance, but degrades video tracking and camera-pose estimation performance, revealing an important tradeoff between semantic and geometric representation learning. These findings suggest that combining image and video SSL objectives can be beneficial in resource-limited settings, while highlighting the need for improved methods that better balance semantic, temporal, and geometric supervision.
