Search papers, labs, and topics across Lattice.
This paper introduces a CLIP-guided label-free region scoring framework that evaluates various scoring strategies for fine-grained classification without relying on ground-truth labels. By comparing cosine similarity, margin-based, and entropy-based scoring methods using both SAM-generated masks and random crops across five datasets, the study finds that Soft Negative Margin scoring yields the best results, while random crops outperform SAM masks due to their ability to maintain contextual information. The findings highlight the effectiveness of pseudo-label scoring in approximating true-label performance, offering new insights into the selection of discriminative evidence in fine-grained classification tasks.
Random crop-based pseudo-label scoring consistently outperforms SAM-generated masks, revealing a surprising advantage in preserving contextual information for fine-grained classification.
Recent vision models such as CLIP and SAM enable training-free segmentation and semantic encoding for fine-grained classification. A common approach is to compare the representations of segmented image regions with the text prompt embeddings of the corresponding labels. However, it remains unclear how different local regions and CLIP-based scoring strategies affect the selection of discriminative evidence, especially when ground-truth labels are unavailable. In this paper, we propose a unified CLIP-guided label-free region scoring framework for fine-grained classification. The framework evaluates cosine similarity-based, margin-based, and entropy-based scoring strategies using both SAM-generated masks and random crops, and introduces two label-free pseudo-label variants based on global image embeddings and local region embeddings. We conduct experiments on five fine-grained classification datasets to systematically compare different region generation methods and scoring strategies. The results show that Soft Negative Margin scoring achieves the strongest performance, and pseudo-label scoring closely approximates true-label performance. Although SAM produces semantically meaningful masks, random-crop-based pseudo-label scoring consistently outperforms SAM-based scoring across all datasets, suggesting that random crops preserve surrounding information and provide more stable semantic context when pseudo-labels are noisy. In addition, SAM masks benefit from aggregating embeddings from all regions, whereas random crops tend to perform better with a smaller top-k subset. These findings provide new insights for fine-grained classification.
