Search papers, labs, and topics across Lattice.
This paper introduces Image-Disambiguated Video Temporal Grounding (ID-VTG), a novel task that enhances Video Temporal Grounding by integrating multimodal queries consisting of reference images and text descriptions to accurately localize actions involving visually similar entities. The authors construct two benchmarks, IDVTG-Gym and IDVTG-InternVid, to evaluate their method, which employs a dual-branch architecture called Visually-Guided Disambiguation Aggregation (VGD-Agg) to improve event proposal generation and frame-level matching. Experimental results demonstrate that ID-VTG achieves state-of-the-art performance on these benchmarks, highlighting its effectiveness in resolving ambiguities in video grounding tasks.
By combining images with text queries, ID-VTG significantly improves the accuracy of video grounding in scenarios with visually similar entities.
Video Temporal Grounding (VTG) faces significant challenges when natural language queries must distinguish between multiple events involving visually similar entities, particularly when relying on fine-grained visual attributes that are difficult to describe accurately in words alone. To address this, we introduce Image-Disambiguated Video Temporal Grounding (ID-VTG), a task that leverages multimodal queries combining a reference image and a text description to precisely localize segments where a specific instance performs a described action. To facilitate research, we construct two benchmarks: IDVTG-Gym, focusing on fine-grained, compositionally ordered gymnastics actions with athletes in similar uniforms; and IDVTG-InternVid, an open-world dataset featuring diverse entities (e.g., humans, animals, fictional characters) and significant temporal distractors. Methodologically, we propose the Visually-Guided Disambiguation Aggregation (VGD-Agg) framework based on a dual-branch fast-slow architecture. The fast branch efficiently generates preliminary event proposals, while the slow branch performs fine-grained frame-level matching between video frames and the reference image. We enhance discriminability via two learnable tokens: a Compare Token, which represents hard negatives to probe for the presence of the target instance (as referred to by the query image), and a Depress Value, which represents text-irrelevant events. Proposals that the Compare Token identifies as lacking the target instance are pushed toward the Depress Value, thus easing disambiguation via the text query. Extensive experiments validate our approach, which achieves state-of-the-art results on the proposed benchmarks. Code is available at https://github.com/oceanflowlab/ID-VTG.
Tsinghua AI