Search papers, labs, and topics across Lattice.
VideoStir addresses the challenge of limited context windows in multimodal LLMs for long videos by introducing a structured and intent-aware retrieval-augmented generation (RAG) framework. It constructs a spatio-temporal graph of video clips and performs multi-hop retrieval to capture contextually related events, while also employing an MLLM-backed intent-relevance scorer to align retrieved frames with the query's reasoning intent. Experiments demonstrate that VideoStir achieves competitive performance compared to state-of-the-art baselines without relying on auxiliary information.
Forget flattening: VideoStir's spatio-temporal graph retrieval and intent-aware scoring unlocks more effective reasoning over long videos.
Scaling multimodal large language models (MLLMs) to long videos is constrained by limited context windows. While retrieval-augmented generation (RAG) is a promising remedy by organizing query-relevant visual evidence into a compact context, most existing methods (i) flatten videos into independent segments, breaking their inherent spatio-temporal structure, and (ii) depend on explicit semantic matching, which can miss cues that are implicitly relevant to the query's intent. To overcome these limitations, we propose VideoStir, a structured and intent-aware long-video RAG framework. It firstly structures a video as a spatio-temporal graph at clip level, and then performs multi-hop retrieval to aggregate evidence across distant yet contextually related events. Furthermore, it introduces an MLLM-backed intent-relevance scorer that retrieves frames based on their alignment with the query's reasoning intent. To support this capability, we curate IR-600K, a large-scale dataset tailored for learning frame-query intent alignment. Experiments show that VideoStir is competitive with state-of-the-art baselines without relying on auxiliary information, highlighting the promise of shifting long-video RAG from flattened semantic matching to structured, intent-aware reasoning. Codes and checkpoints are available at Github.
