Search papers, labs, and topics across Lattice.
This paper introduces WSRVOS, a novel weakly-supervised method for referring video object segmentation that utilizes only text expressions as supervision, eliminating the need for costly pixel-level annotations. By employing a contrastive referring expression augmentation scheme and bi-directional vision-language feature selection, the model achieves fine-grained multimodal alignment, enhancing its ability to distinguish between positive and negative expressions. Extensive experiments across four datasets demonstrate that WSRVOS significantly outperforms existing methods, showcasing its effectiveness in weakly-supervised settings.
Training video object segmentation models with just text expressions could revolutionize the field by drastically reducing the need for expensive annotations.
Referring video object segmentation (RVOS) aims to segment the target instance in a video, referred by a text expression. Conventional approaches are mostly supervised learning, requiring expensive pixel-level mask annotations. To tackle it, weakly-supervised RVOS has recently been proposed to replace mask annotations with bounding boxes or points, which are however still costly and labor-intensive. In this paper, we design a novel weakly-supervised RVOS method, namely WSRVOS, to train the model with only text expressions. Given an input video and the referring expression, we first design a contrastive referring expression augmentation scheme that leverages the captioning capabilities of a multimodal large language model to generate both positive and negative expressions. We extract visual and linguistic features from the input video and generated expressions, then perform bi-directional vision-language feature selection and interaction to enable fine-grained multimodal alignment. Next, we propose an instance-aware expression classification scheme to optimize the model in distinguishing positive from negative expressions. Also, we introduce a positive-prediction fusion strategy to generate high-quality pseudo-masks, which serve as additional supervision to the model. Last, we design a temporal segment ranking constraint such that the overlaps between mask predictions of temporally neighboring frames are required to conform to specific orders. Extensive experiments on four publicly available RVOS datasets, including A2D Sentences, J-HMDB Sentences, Ref-YouTube-VOS, and Ref-DAVIS17, demonstrate the superiority of our method. Code is available at \href{https://github.com/viscom-tongji/WSRVOS}{https://github.com/viscom-tongji/WSRVOS}.
