Search papers, labs, and topics across Lattice.
This paper introduces Structured-Condensed Prompt Tuning (SCPT) for enhancing fine-grained image recognition in vision-language models (VLMs) by addressing the limitations of traditional prompt tuning methods that treat class labels as isolated entities. By employing Semantic Relation Encoding (SRE) to model inter-class semantic relationships and a Semantic Condensation loss (ScLoss) to refine supervision, SCPT significantly improves the model's ability to capture hierarchical dependencies and inter-class correlations. Experimental results across 14 fine-grained benchmarks demonstrate that SCPT achieves state-of-the-art performance, effectively reducing semantic ambiguity and enhancing discrimination in both few-shot and base-to-novel scenarios.
Fine-grained image recognition can be revolutionized by a novel approach that captures complex semantic relationships, achieving state-of-the-art results with minimal labeled data.
Fine-grained image recognition poses a significant challenge due to the substantial expertise and effort required for manual annotation. Vision-language models (VLMs) like CLIP provide a compelling zero-shot alternative, reducing reliance on extensive labeled data. However, their ability to capture subtle distinctions remains limited, leading to subpar recognition performance. While prompt tuning has proven effective for adapting VLMs, most existing methods treat class labels as isolated, discrete entities, overlooking the rich semantic relationships between them. This oversimplified assumption limits the model's ability to capture hierarchical dependencies and inter-class correlations -- both critical for distinguishing visually similar categories. The problem is especially acute in fine-grained classification, where accurate recognition depends on understanding complex label semantics. To address this, we propose Structured-Condensed Prompt Tuning (SCPT), which enhances semantic structure modeling in prompt learning. Specifically, we introduce Semantic Relation Encoding (SRE) to explicitly model inter-class semantic topology and encode structured label relationships. In parallel, we design a Semantic Condensation loss (ScLoss) to suppress redundant supervision and extract discriminative components from the global semantic space. Together, these components significantly improve semantic alignment and fine-grained discrimination. Extensive experiments on 14 fine-grained benchmarks show that SCPT effectively mitigates semantic ambiguity and achieves state-of-the-art performance in both few-shot and base-to-novel generalization settings.
