Search papers, labs, and topics across Lattice.
The paper introduces GraphThinker, a reinforcement learning-based approach to improve video reasoning by explicitly modeling causal relationships between events using event-based video scene graphs (EVSGs). GraphThinker leverages an MLLM to construct EVSGs capturing intra- and inter-event relations and incorporates these graphs as an intermediate reasoning step. By introducing a visual attention reward during reinforcement finetuning, the method strengthens visual grounding and reduces hallucinations.
By explicitly modeling event-level causal relationships in videos via scene graphs, GraphThinker significantly reduces hallucinations in video reasoning tasks.
Video reasoning requires understanding the causal relationships between events in a video. However, such relationships are often implicit and costly to annotate manually. While existing multimodal large language models (MLLMs) often infer event relations through dense captions or video summaries for video reasoning, such modeling still lacks causal understanding. Without explicit causal structure modeling within and across video events, these models suffer from hallucinations during the video reasoning. In this work, we propose GraphThinker, a reinforcement finetuning-based method that constructs structural event-level scene graphs and enhances visual grounding to jointly reduce hallucinations in video reasoning. Specifically, we first employ an MLLM to construct an event-based video scene graph (EVSG) that explicitly models both intra- and inter-event relations, and incorporate these formed scene graphs into the MLLM as an intermediate thinking process. We also introduce a visual attention reward during reinforcement finetuning, which strengthens video grounding and further mitigates hallucinations. We evaluate GraphThinker on two datasets, RexTime and VidHalluc, where it shows superior ability to capture object and event relations with more precise event localization, reducing hallucinations in video reasoning compared to prior methods.
