Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper tackles the challenges of open-vocabulary instance segmentation (OVIS) and open-set panoptic segmentation (OSPS) by introducing a multimodal framework that generates robust pseudo-labels using pre-trained vision-language models. The approach addresses issues related to noisy pseudo-masks and the handling of synonyms and out-of-vocabulary words through CLIP-guided filtering and GPT-based caption reconstruction. Experimental results on the COCO dataset show that the proposed method significantly outperforms existing state-of-the-art techniques, achieving notable advancements in both OVIS and OSPS tasks.
Leveraging multimodal frameworks, this approach achieves substantial improvements in segmentation tasks by effectively generating robust pseudo-labels without manual annotation.
This work addresses the challenge of open-vocabulary instance segmentation (OVIS) and open-set panoptic segmentation (OSPS), which aim to recognize both predefined and unseen object categories without exhaustive human annotations. Existing methods often suffer from noisy pseudo-masks, limited visual-textual grounding, and difficulty handling synonyms or out-of-vocabulary (OOV) words. To overcome these challenges, we propose a multimodal framework that leverages pre-trained vision-language models for automatic pseudo-label generation, CLIP-guided synonym filtering, and GPT-based caption reconstruction. In our target-vocabulary-assisted pseudo-labeling setting, the framework first constructs pseudo segmentation masks, descriptive captions, and semantically aligned synonym sets using Grounded SAM, LLaVA, and CLIP, providing multimodal supervision without manual annotation. We then enhance visual-textual alignment through three complementary training objectives: an extended grounding loss that incorporates visually grounded synonyms, a semantic consistency loss, and a generative caption reconstruction loss. Extensive experiments on the COCO dataset demonstrate that the proposed method consistently outperforms previous state-of-the-art approaches under this protocol, achieving substantial improvements on both OVIS and OSPS benchmarks.