Search papers, labs, and topics across Lattice.
This paper introduces SFF-CLIP, a novel fine-tuning approach for the CLIP model that enhances its fine-grained dense feature representation without requiring additional region annotations. By leveraging self-annotated image-text pairs and a run-time region-phrase alignment scheme, the method aligns concept phrases with region-based features while preserving global visual-semantic consistency. Extensive experiments show that SFF-CLIP significantly improves fine-grained representation performance while maintaining the original CLIP's effectiveness on image-level tasks.
SFF-CLIP boosts fine-grained feature representation in CLIP without the need for cumbersome region annotations, achieving significant performance gains.
Contrastive Language-Image Pre-training (CLIP) has been shown to have limitations in its fine-grained dense feature representation, due to its pre-training focusing on matching the whole image to a text description. Considering the large data and computational burden in pre-training a vision-language model from scratch, a series of works aim to enhance the fine-grained ability of CLIP through a fine-tuning scheme. However, existing works suffer from a variety of limitations: additional region annotations are usually required, which limits the semantic diversity due to the predefined categories and leads to a large effort to process the training data; and they usually sacrifice CLIP's original ability for global visual representation. To bypass these limitations, we propose SFF-CLIP (Self-annotated Fine-grained Fine-tuning for CLIP), which only uses image-text pairs as input to boost the fine-grained representation ability in the CLIP fine-tuning, while maintaining the global visual-semantic consistency. Concretely, a run-time region-phrase alignment scheme is designed, which obtains concept phrases from the input sentence, and aligns them with corresponding extracted region-based features using text-specific heat maps. Extensive experiments demonstrate that SFF-CLIP leads to significant performance improvements on fine-grained dense feature representation, as well as maintaining the performance of the original CLIP on image-level tasks. Code will be released later.
