Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for explaining image similarity by utilizing Concept Activation Vectors (CAVs) derived from Sparse Autoencoders (SAEs), enabling a deeper understanding of the factors driving similarity scores. By perturbing image embeddings along identified concept directions, the authors measure the impact on similarity functions, providing both local and global insights into image similarity. The results demonstrate that this approach yields more faithful representations of data distribution compared to traditional pixel-space methods, enhancing interpretability in both individual and group contexts.
Understanding image similarity just got clearer: this framework reveals how specific concepts like texture and shape drive similarity scores, outperforming traditional methods.
Image similarity underlies many computer vision applications, yet it is often unclear why two images receive a high or low similarity score. Existing explainability methods often rely on gradient-based attribution maps to provide local justifications for similarity. These approaches struggle to provide global insights into what specifically drives similarity in regions of an embedding space, such as texture, shape, or color. We introduce a model- and metric-agnostic framework that explains image similarity using Concept Activation Vectors (CAVs) extracted automatically via Sparse Autoencoders (SAEs). Given a pair of images, we perturb their embeddings along discovered concept directions and measure the resulting change in a chosen similarity function, yielding concept importances. For image pairs, we provide localization with concept attribution maps. We extend this procedure to group-level settings, explaining what drives similarity across a cluster of images rather than a single pair, and further, we introduce Exemplar Retrieval, aiming to recover samples with similar reasons contributing to similarity. Our experiments show that our latent perturbations are more faithful to the underlying data distribution than pixel-space baselines, and that concept importances linearly recover the true similarity score. Qualitative results further confirm the usefulness of our methods in understanding a model's individual and group similarity judgments.
