Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces a visualization protocol based on unsupervised semantic segmentation to facilitate the understanding of self-supervised vision transformers (ViTs). By visualizing segmentation results, the protocol reveals model behaviors that consistently emerge across images, such as positional biases and scaling behaviors. Applying this protocol to a diverse set of SSL models uncovers novel insights, including boundary artifacts in DINOv3-Large, and distinguishes between positional effects and locality bias.
See how unsupervised segmentation reveals surprising positional biases and boundary artifacts in even state-of-the-art self-supervised ViTs like DINOv3-Large.
Self-supervised learning (SSL) has produced a diverse landscape of vision transformers (ViTs) whose pretrained representations support a wide range of downstream tasks. Towards a better understanding of these models, a body of work has assessed the mechanics of their self-attention as well as the types of information captured across their representations, revealing, for example, stark differences between models trained with contrastive learning (CL) and masked image modeling (MIM). However, these advances in model understanding have not yet fully permeated the broader community, where insights specific to CL models are sometimes generalized to MIM models. To make model understanding straightforward and intuitive for a broad audience, we propose a simple and easily interpretable visualization protocol. Our protocol is based on visualizing unsupervised semantic segmentation results, yet our goal is not to maximize segmentation performance. Instead, it allows us to convey model behaviors that consistently emerge across images. Benchmarking a diverse set of SSL models across layers and representations, we obtain novel insights into distinct positional biases and scaling behaviors, including strong boundary artifacts in DINOv3-Large model tokens. These insights complement and help communicate a range of previous findings. Our protocol further enables a clear visual distinction between positional effects and the closely related but distinct locality bias, which has been studied much more extensively in the literature. The protocol is publicly available on GitHub and we believe it will catalyze further model understanding for a broad community.