Search papers, labs, and topics across Lattice.
VisionCoach is introduced as an input-adaptive reinforcement learning framework that enhances spatio-temporal grounding in video reasoning by selectively applying visual prompts during training to amplify relevant evidence and suppress distractors. A Visual Prompt Selector predicts appropriate prompt types conditioned on the video and question, while a Spatio-Temporal Reasoner is optimized with RL under visual prompt guidance and object-aware grounding rewards. Experiments across diverse video benchmarks demonstrate state-of-the-art performance with a single efficient inference pathway, showing that visual prompting during training improves grounded video reasoning, which is internalized via self-distillation for prompt-free inference.
Achieve state-of-the-art video reasoning by using visual prompting during training to guide reinforcement learning, then distilling this ability into a model that performs grounded reasoning on raw videos without prompts at inference time.
Video reasoning requires models to locate and track question-relevant evidence across frames. While reinforcement learning (RL) with verifiable rewards improves accuracy, it still struggles to achieve reliable spatio-temporal grounding during the reasoning process. Moreover, improving grounding typically relies on scaled training data or inference-time perception tools, which increases annotation cost or computational cost. To address this challenge, we propose VisonCoach, an input-adaptive RL framework that improves spatio-temporal grounding through visual prompting as training-time guidance. During RL training, visual prompts are selectively applied to challenging inputs to amplify question-relevant evidence and suppress distractors. The model then internalizes these improvements through self-distillation, enabling grounded reasoning directly on raw videos without visual prompting at inference. VisonCoach consists of two components: (1) Visual Prompt Selector, which predicts appropriate prompt types conditioned on the video and question, and (2) Spatio-Temporal Reasoner, optimized with RL under visual prompt guidance and object-aware grounding rewards that enforce object identity consistency and multi-region bounding-box overlap. Extensive experiments demonstrate that VisonCoach achieves state-of-the-art performance under comparable settings, across diverse video reasoning, video understanding, and temporal grounding benchmarks (V-STAR, VideoMME, World-Sense, VideoMMMU, PerceptionTest, and Charades-STA), while maintaining a single efficient inference pathway without external tools. Our results show that visual prompting during training improves grounded video reasoning, while self-distillation enables the model to internalize this ability without requiring prompts at inference time.
