Search papers, labs, and topics across Lattice.
The paper introduces Video-ToC, a video reasoning framework that uses tree-of-cue reasoning to improve video understanding in Video LLMs. It incorporates a tree-guided visual cue localization, a reasoning-demand reward mechanism for reinforcement learning, and two new datasets (Video-ToC-SFT-1k and Video-ToC-RL-2k) for training. Experiments on seven benchmarks show Video-ToC outperforms existing methods in video understanding and reduces hallucinations.
Video-ToC drastically improves video understanding by forcing Video LLMs to focus on relevant visual cues, leading to state-of-the-art performance and reduced hallucinations.
Existing Video Large Language Models (Video LLMs) struggle with complex video understanding, exhibiting limited reasoning capabilities and potential hallucinations. In particular, these methods tend to perform reasoning solely relying on the pretrained inherent reasoning rationales whilst lacking perception-aware adaptation to the input video content. To address this, we propose \textbf{Video-ToC}, a novel video reasoning framework that enhances video understanding through tree-of-cue reasoning. Specifically, our approach introduces three key innovations: (1) A tree-guided visual cue localization mechanism, which endows the model with enhanced fine-grained perceptual capabilities through structured reasoning patterns; (2) A reasoning-demand reward mechanism, which dynamically adjusts the reward value for reinforcement learning (RL) based on the estimation of reasoning demands, enabling on-demand incentives for more effective reasoning strategies; and (3) An automated annotation pipeline that constructs the Video-ToC-SFT-1k and Video-ToC-RL-2k datasets for supervised fine-tuning (SFT) and RL training, respectively. Extensive evaluations on six video understanding benchmarks and a video hallucination benchmark demonstrate the superiority of Video-ToC over baselines and recent methods. Code is available at https://github.com/qizhongtan/Video-ToC.
