Search papers, labs, and topics across Lattice.
This paper introduces an unsupervised framework that adapts vision-language models (VLMs) to enhance multi-label image recognition without requiring labeled data. By employing a two-stage approach鈥攃utting and sewing鈥攖he method mitigates the models' tendency to focus on iconic objects, allowing for a more inclusive understanding of images. Experimental results demonstrate that this framework significantly outperforms existing unsupervised methods and even weakly supervised baselines across four public datasets, showcasing its effectiveness in achieving comprehensive visual understanding.
Vision-language models can be transformed to recognize multiple objects in images without any labels, challenging the conventional focus on iconic representations.
Understanding multi-label images remains a challenging task in computer vision. With the rapid progress of vision-language multimodal learning, vision-language models (VLMs) enable zero-shot recognition without labeled data. However, due to their intrinsic design, these models often prioritize the most iconic object and omit other contextual positives. This intrinsic bias conflicts with the nature of multi-label learning, thereby limiting their applicability. In this work, we propose an unsupervised framework that adapts VLMs from iconic recognition toward inclusive understanding, enabling label-free multi-label image recognition. Our approach consists of two key stages, ``cutting'' and ``sewing'': In the cutting stage, we present the multi-sampling response estimator to prevent the model from concentrating only on one single object. In the second sewing stage, the multi-object blend adaptation is introduced to adjust the labels to better conform to the multi-label distribution while preserving the intrinsic characteristics of the original model within only one epoch. Extensive experiments show that our framework significantly outperforms existing unsupervised approaches on four public datasets, even surpassing several representative weakly supervised baselines. These results demonstrate the potential of adapting pre-trained VLMs for more comprehensive visual understanding without manual annotations. Our code is publicly available at https://github.com/iCVTEAM/TailorCLIP.
