Search papers, labs, and topics across Lattice.
This paper tackles the problem of poor performance of Referring Image Segmentation (RIS) models on motion-related queries. They introduce a data augmentation technique to generate motion-centric phrases from existing captions and propose Multimodal Radial Contrastive Learning (MRaCL) on fused image-text embeddings to better capture contextual dependencies. Experiments on a new motion-centric test split and benchmark (M-Bench) demonstrate significant performance gains on motion-based queries while maintaining competitive accuracy on appearance-based descriptions.
RIS models struggle with motion-based queries, but a new data augmentation and contrastive learning approach closes the gap without sacrificing performance on appearance-based descriptions.
Referring Image Segmentation (RIS) requires identifying objects from images based on textual descriptions. We observe that existing methods significantly underperform on motion-related queries compared to appearance-based ones. To address this, we first introduce an efficient data augmentation scheme that extracts motion-centric phrases from original captions, exposing models to more motion expressions without additional annotations. Second, since the same object can be described differently depending on the context, we propose Multimodal Radial Contrastive Learning (MRaCL), performed on fused image-text embeddings rather than unimodal representations. For comprehensive evaluation, we introduce a new test split focusing on motion-centric queries, and introduce a new benchmark called M-Bench, where objects are distinguished primarily by actions. Extensive experiments show our method substantially improves performance on motion-centric queries across multiple RIS models, maintaining competitive results on appearance-based descriptions. Codes are available at https://github.com/snuviplab/MRaCL
