Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the impact of self-distillation on learning dynamics by training a group of randomly initialized networks without the typical complex components found in state-of-the-art self-supervised methods. The authors find that even in this minimalistic setup, the networks achieve significant improvements in learned representations compared to a random baseline on downstream tasks. Additionally, the research explores how variations in hyperparameters influence the learning outcomes, providing insights into the underlying mechanisms at play.
Removing complex components from self-supervised learning reveals that even simple, randomly initialized networks can significantly outperform random baselines through peer-to-peer consensus.
In self-supervised learning, self-distilled methods have shown impressive performance, learning representations useful for downstream tasks and even displaying emergent properties. However, state-of-the-art methods usually rely on ensembles of complex mechanisms, with many design choices that are empirically motivated and not well understood. In this work, we explore the role of self-distillation within learning dynamics. Specifically, we isolate the effect of self-distillation by training a group of randomly initialized networks, removing all other common components such as projectors, predictors, and even pretext tasks. Our findings show that even this minimal setup can lead to learned representations with non-trivial improvements over a random baseline on downstream tasks. We also demonstrate how this effect varies with different hyperparameters and present a short analysis of what is being learned by the models under this setup.