Search papers, labs, and topics across Lattice.
This paper introduces DART, a novel approach for zero-shot video temporal grounding that addresses the reasoning gap in existing methods by integrating difficulty-aware routing with structured reasoning in vision-language models. By employing a query-conditioned Determinantal Point Process (DPP), DART effectively selects relevant keyframes and adapts its processing path based on query complexity, allowing for more nuanced understanding of temporal ordering and causal relationships. The method achieves state-of-the-art performance on benchmark datasets, improving mean Intersection over Union (mIoU) by up to 3.5 points while utilizing significantly fewer frames than previous approaches.
DART bridges the reasoning gap in zero-shot video temporal grounding, achieving a remarkable 3.5-point mIoU improvement with over 7 times fewer frames.
Zero-shot video temporal grounding (VTG) localizes events in untrimmed videos from natural language queries without task-specific training. Existing methods rely on frame-query feature matching, which suffices for simple events but struggles with complex multi-stage queries that require understanding temporal ordering and causal structure -- a disparity we call the reasoning gap. We propose DART (Difficulty-Adaptive Routing for Temporal Grounding), which bridges this gap by coupling difficulty-aware routing with structured reasoning in large vision-language models. A query-conditioned Determinantal Point Process (DPP) serves a dual role: selecting diverse, query-relevant keyframes as temporal evidence, and providing spectral entropy as a difficulty indicator. Simple queries are routed to a Fast path for direct prediction, while complex queries follow a Slow path with Temporal Markup Prompting, which decomposes localization into global event analysis, per-frame temporal role annotation, and boundary extraction. On Charades-STA and ActivityNet Captions, DART achieves state-of-the-art zero-shot performance across both identically distributed and multiple out-of-distribution settings, improving mIoU by up to 3.5 points over the strongest baseline while using over 7 times fewer frames. The project homepage is available at https://dart-vtg.github.io/.
