Search papers, labs, and topics across Lattice.
This paper addresses the limitations of existing perceptual similarity metrics by introducing a large-scale dataset of human similarity judgments over image triplets, annotated across multiple semantic aspects. The authors benchmark various vision-language models (VLMs) against this dataset and find a significant performance gap compared to human consensus. They then fine-tune a VLM to create the Text-Prompted Image Perceptual Similarity (TPIPS) metric, which better captures human-like visual similarity and enhances capabilities in text-guided retrieval and generative model evaluation.
TPIPS reveals that traditional metrics fail to capture the nuanced aspects of visual similarity, leading to a performance gap that can be bridged with context-aware evaluation.
Human visual similarity judgments are context-dependent. For example, two images may be similar in shape but distinct in color. Existing perceptual similarity metrics, however, collapse these nuances into a single scalar value, offering no mechanism to condition on specific aspects. To bridge this gap, we introduce a large-scale dataset of human similarity judgments over image triplets, where each triplet is annotated across multiple, free-form semantic aspects of similarity. Benchmarking a broad range of frontier vision-language models (VLMs) reveals a considerable performance gap compared to human annotators'consensus. Leveraging our data, we fine-tune a VLM to produce our Text-Prompted Image Perceptual Similarity (TPIPS) metric, capturing multiple senses of visual similarity depending on the specified text prompt. We demonstrate that TPIPS aligns more closely with human perception and generalizes reliably beyond the training distribution. Finally, we show that TPIPS unlocks new capabilities in text-guided retrieval, compositional search, and the fine-grained evaluation of generative models. Our code, data, and trained models are at https://peterwang512.github.io/TPIPS
BAIR