Search papers, labs, and topics across Lattice.
This paper introduces LeVLJEPA, a novel non-contrastive end-to-end vision-language pretraining method that leverages cross-modal prediction and distributional regularization without relying on negatives or complex training schedules. The approach enables the encoder to produce superior dense semantic features, outperforming traditional contrastive methods in tasks such as GQA, VQAv2, and POPE while maintaining competitive performance in global readouts. These findings highlight the potential of non-contrastive pretraining to enhance the effectiveness of vision-language models in downstream applications.
Non-contrastive pretraining can yield dense semantic features that outperform traditional contrastive methods in vision-language tasks.
Vision-language pretraining remains dominated by contrastive objectives, whereas vision-only self-supervised learning has largely adopted non-contrastive methods. At the same time, the role of vision-language encoders has shifted: they are increasingly deployed not as zero-shot classifiers but as the frozen visual backbone of vision-language models and dense prediction systems, which consume the full grid of patch tokens rather than a single pooled embedding. We introduce LeVLJEPA, the first fully non-contrastive end-to-end vision-language pretraining method. LeVLJEPA learns through cross-modal prediction with stop-gradient targets and per-modality distributional regularization, without negatives, temperature, momentum encoder, or teacher-student schedule, and trains stably at large scale. We find that the resulting encoder provides markedly stronger dense semantic features for downstream use: as a frozen vision-language-model backbone, LeVLJEPA is the strongest of the evaluated encoders across GQA, VQAv2, and POPE under two distinct language models, and outperforms contrastive baselines on semantic segmentation, while remaining on par on global readouts such as linear probing. These results establish non-contrastive pretraining as an effective means of producing dense semantic vision features.
