Search papers, labs, and topics across Lattice.
This paper introduces REZE, a recognition-based method for video temporal grounding that operates without training by leveraging clip-level confidence scores from a vision-language model. By converting these scores into time intervals using a deterministic algorithm, REZE adapts to various output requirements, achieving significant improvements in moment retrieval and highlight detection tasks. The method sets new benchmarks for training-free approaches, surpassing previous state-of-the-art results on multiple datasets, including QVHighlights and Charades-STA.
REZE achieves a new state of the art in training-free video temporal grounding, outperforming fully supervised models in highlight detection and moment retrieval.
Video temporal grounding (VTG) refers to the task of identifying the time interval in a video that corresponds to a given natural-language query. A common zero-shot strategy asks a large vision-language model (VLM) to generate the start and end timestamps directly, so the result depends heavily on the design and training of the model, and grounding accuracy differs widely from one VLM to another. We therefore propose REcognition-based Zero-shot Extraction (REZE), a simple training-free method that splits the video into short clips, asks the model for a clip-level confidence score for the query, and uses a deterministic algorithm to convert the resulting score curve into the output required by the task. Because temporal aggregation is performed outside the model, REZE adapts to different task outputs, from single- and multi-interval moment retrieval to highlight detection. On QVHighlights, REZE improves the best reported training-free moment-retrieval mAP from 38.23 to 40.32, while on highlight detection it reaches 44.18 mAP and 73.41 HIT@1, establishing a new state of the art among training-free methods. Its HIT@1 also outperforms all fully supervised SoTAs on the QVHighlights test split. We evaluate REZE on seven backbones from three model families. On Charades-STA and QVHighlights, it outperforms direct timestamp generation in every available comparison. We further observe that with REZE an earlier-generation model can approach the native performance of a newer model in its family.
Tsinghua AI