Search papers, labs, and topics across Lattice.
This paper introduces the Expression-driven Motion Calibration (EMC) framework for Referring Video Object Segmentation (RVOS), which addresses the limitations of existing methods by explicitly modeling the dependency of language expressions on motion semantics. The EMC framework utilizes a Motion Signal Processing (MSP) module to extract motion control signals and a Motion Influence Calibration (MIC) module to adaptively adjust the impact of motion cues during segmentation tasks. Extensive evaluations across six benchmarks demonstrate that this approach significantly enhances segmentation accuracy by effectively integrating motion information tailored to specific expressions.
Unlocking motion semantics in language expressions boosts video object segmentation accuracy, revealing a critical link between language and motion processing.
Referring Video Object Segmentation (RVOS) aims to segment referred objects at the pixel level in video sequences based on natural language descriptions. Existing methods typically introduce motion information within a unified cross-modal temporal modeling framework, where language cues are used for target localization and segmentation. However, the dependency of expressions on motion semantics is not explicitly modeled, making it difficult to adaptively adjust the use of motion information according to different semantic requirements. To address these issues, we propose an Expression-driven Motion Calibration (EMC) framework for RVOS that explicitly unlocks and leverages the motion semantics within expressions. The proposed method extracts interpretable motion control signals from expressions via a Motion Signal Processing (MSP) module, and employs a Motion Influence Calibration (MIC) module to adjust the contribution of motion cues during temporal decision making. In addition, a Semantic Temporal Stage Construction (STSC) module is introduced to build expression-relevant temporal stages, providing a compact temporal candidate space for motion calibration. Through extensive evaluation on six standard benchmarks, including Ref-YouTubeVOS, Ref-DAVIS17, MeViS (valid/valid$^u$), A2D-Sentences, and JHMDB-Sentences, the superiority of our method is validated. We will release the code on https://github.com/Jeven7/EMC.
