Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to zero-shot video moment retrieval (ZMR) that addresses the challenges posed by modality and language-style gaps in existing methods. By leveraging a Self-Similarity-based Moment Proposal and Scoring mechanism, the authors exploit intrinsic relationships within video content, leading to more reliable span generation and scoring without relying on potentially noisy query-to-video similarities. Extensive experiments show that their method, Self-SiMS, achieves state-of-the-art performance across various ZMR benchmarks, highlighting its effectiveness in improving retrieval accuracy.
Self-SiMS redefines zero-shot video moment retrieval by eliminating reliance on noisy query similarities, achieving unprecedented accuracy in span generation.
Zero-shot video moment retrieval aims to overcome the limitations of traditional approaches that require large-scale datasets annotated with text and its relevant temporal spans. Despite advances in pre-trained vision-language models and multimodal large language models, existing ZMR methods still heavily depend on query-to-video content similarity, making them vulnerable to modality and language-style gaps. These gaps lead to unreliable span proposals and unstable moment retrieval results. To address this issue, we propose Self-Similarity-based Moment Proposal and Scoring that instead exploits intrinsic relationships within videos, enabling robust span generation and scoring. By deriving self-similarity only from the video content, we circumvent the noisy and mismatched patterns of query-frame or query-caption similarities, thereby mitigating both modality and language-style gaps. Furthermore, we introduce a query-aware MLLM-based reasoning stage to further sharpen alignment between text and video. Extensive experiments demonstrate that Self-SiMS achieves state-of-the-art performance across ZMR benchmarks.
