Search papers, labs, and topics across Lattice.
This paper introduces ViTexQA, a novel video-text question answering dataset designed to enhance multimodal understanding by addressing the challenges of integrating temporally distributed textual cues across multiple frames. The authors developed a quality-controlled Chain-of-Thought annotation pipeline with temporal constraints to ensure that all questions require cross-frame text fusion, thereby reflecting real-world comprehension needs. Their approach, FrameThinker, employs a two-stage training process that combines supervised fine-tuning with reinforcement learning, resulting in a significant performance improvement over state-of-the-art baselines, as evidenced by a 6.3% increase in ROUGE-L scores.
Multi-frame temporal reasoning is crucial for video-text understanding, and ViTexQA reveals that existing datasets fall short, leaving a significant gap in multimodal comprehension.
Despite remarkable progress in multimodal understanding, current MLLMs still exhibit limitations in video text understanding, particularly when semantics emerge through the integration of temporally distributed textual cues across multiple frames. This perception challenge fundamentally differs from static image text understanding, yet existing datasets fail to capture: the vast majority of questions remain answerable from single frames, inadequately reflecting real-world video text comprehension demands. To address this, we present ViTexQA, a large-scale video-text QA dataset, and FrameThinker for robust multi-frame temporal reasoning. We build ViTexQA via a quality-controlled Chain-of-Thought (CoT) annotation pipeline boosted with temporal constraints; all its QA pairs demand cross-frame text fusion to solve, enforcing true temporal reliance. FrameThinker adopts two-stage training for explicit temporal modeling: CoT-Guided Supervised Fine-Tuning (SFT) generates frame-aware reasoning chains, followed by Temporally-grounded Reinforcement Learning (RL) optimized with multi-frame coherence rewards. Evaluations show our method outperforms SOTA baselines on ViTexQA, lifting ROUGE-L by 6.3%.
Tsinghua AI