Search papers, labs, and topics across Lattice.
This paper details a staged approach for audio-visual segmentation in the MeViS-Audio track, where a system segments objects based on spoken motion expressions in videos. By leveraging Qwen3-ASR for speech-to-text conversion and employing multiple grounding and segmentation models, the method selects the most reliable mask predictions through agreement among candidates. The final system achieved a top ranking with impressive metrics, including a J & F score of 0.5952 and a target accuracy of 0.9205, demonstrating the effectiveness of agreement-based selection in complex audio-visual tasks.
Achieving a top rank in the MeViS-Audio challenge, this system showcases the power of agreement-based segmentation in accurately identifying objects described by spoken expressions.
The MeViS-Audio track asks a system to segment the objects described by a spoken motion expression throughout a video and to return empty masks when the described target is absent. We present a simple staged solution. Qwen3-ASR first converts speech into text. Several video mask tracks are then produced with complementary grounding and segmentation models. Instead of trusting a single prediction, we select the track that has the highest average mask agreement with the other candidates. A small set of explicit direction, count, and plural rules corrects queries that require more than ordinary single-object tracking. Finally, a video-level classifier combines visual, audio-visual, and within-video query scores to decide whether any target is present. The submitted system obtains 0.5952 J &F, 0.7931 no-target accuracy, 0.9205 target accuracy, and a final score of 0.769589. The challenge organizers notified our team that this result ranked first in the track.
