Search papers, labs, and topics across Lattice.
This paper presents Speech2MaskTrack, a novel approach for speech-guided referring video object segmentation that effectively integrates speech recognition with motion-centric temporal grounding and mask tracking. By structuring the spoken query into constraints related to object categories, counts, and interactions, the method enhances the accuracy of object tracking in dynamic scenes. Achieving second place in the MeViS-Audio track of the 8th LSVOS Challenge, the system demonstrates significant advancements in handling complex linguistic instructions for video segmentation tasks.
Speech2MaskTrack reveals how structured speech constraints can dramatically improve object tracking accuracy in video segmentation tasks, even in the absence of acoustic cues.
Speech-guided referring video object segmentation aims to recover the mask tracks of objects specified by a spoken motion description. Here, speech carries a linguistic instruction rather than acoustic evidence from a sounding object, so a solution must connect speech recognition, motion-centric temporal grounding, mask tracking, and explicit no-target handling. We introduce Speech2MaskTrack, our approach for the MeViS-Audio track of the 8th LSVOS Challenge. Speech2MaskTrack transcribes the spoken query and compiles it into structured constraints over category, count, direction, interaction role, and temporal phase. SAM3.1 enumerates multiple instance tracks, which TRACE ranks using complete-trajectory motion and relation evidence. A frozen lexical presence gate may suppress the ranked SAM3.1 base prediction. When the gate predicts that a target is present, an available full-expression-conditioned SaSaSa2VA track replaces the SAM3.1 mask. Only outputs that remain empty enter GPT-assisted recovery, which invokes SaSaSa2VA again under query- and mask-level verification. Speech2MaskTrack achieved second place in the official challenge ranking.
