Search papers, labs, and topics across Lattice.
This paper investigates the role of Siamese student encoders within Joint Embedding Predictive Architectures (JEPA) for self-supervised representation learning, contrasting with traditional single-encoder approaches. By introducing SiamJEPA, which utilizes masked Siamese encoders alongside an exponential moving average teacher network, the authors demonstrate that this architecture enhances representation separability and accelerates early-stage learning. Extensive experiments on ImageNet reveal that SiamJEPA outperforms single-encoder JEPA variants and achieves superior linear probing accuracy compared to Masked Autoencoders, highlighting the significance of Siamese encoders as an inductive bias in predictive representation learning.
Siamese student encoders not only regularize the JEPA objective but also significantly enhance representation learning efficiency and accuracy.
Recently, Joint Embedding Predictive Architectures (JEPAs) have attracted significant attention in the computer vision and machine learning communities as a promising framework for self-supervised representation learning. Unlike masked autoencoders that reconstruct pixels, JEPA models learn representations by predicting latent embeddings of masked regions. Existing JEPA-based methods, such as I-JEPA and V-JEPA, typically employ a single encoder in the student network. In contrast, using Siamese encoders for student network is more naturally aligned with brain-inspired representation learning frameworks, yet their role in JEPA models remains largely unexplored. In this paper, we investigate the effect of Siamese student encoders in JEPA-based representation learning. To this end, we propose SiamJEPA, masked Siamese student encoders equipped with an exponential moving average (EMA) teacher network. SiamJEPA can also be viewed as a JEPA formulation of the brain-inspired representation learning model PhiNet. Through extensive experiments on ImageNet linear probing, we demonstrate that Siamese encoders act as an effective regularizer for the JEPA objective, improving representation separability and accelerating learning during the early stages of training. Furthermore, SiamJEPA consistently outperforms comparable single-encoder JEPA variants under limited training budgets and achieves higher linear probing accuracy than Masked Autoencoders (MAE) which requires longer training. Our findings reveal that Siamese student encoders are not merely an architectural choice but constitute an important inductive bias for predictive representation learning. These results provide new insights into the design of JEPA-based models and suggest that incorporating Siamese student architectures offers a simple yet effective approach for improving self-supervised representation learning.
