Search papers, labs, and topics across Lattice.
This paper introduces the Semantic Evidence Reward (SER), a novel approach to improve spatio-temporal reasoning in video multi-modal language models (MLLMs) by reformulating evidence grounding as a constrained verification task. By utilizing a referee vision-language model (VLM) to assess the relevance and localization quality of generated evidence claims, SER mitigates the limitations of traditional geometry-based rewards that often fail to capture semantic alignment. The method shows significant improvement on the V-STAR benchmark, achieving a 49.6% mLGM score, which is a 3.0-point enhancement over existing evidence-grounded baselines, highlighting its effectiveness in refining both answer accuracy and evidence grounding.
SER's innovative approach to grounding video reasoning demonstrates a 3.0-point leap in accuracy by integrating semantic verification into the reward structure.
Video MLLMs often struggle with fine-grained spatio-temporal reasoning, sometimes generating correct answers based on irrelevant frames or objects. Although outputting spatio-temporal evidence during reasoning is a promising direction, existing RL frameworks typically rely on geometry-only (IoU) rewards, which can be sensitive to boundary perturbations and overlook semantic alignment. To address this, we propose Semantic Evidence Reward (SER), which reformulates spatio-temporal evidence grounding as a constrained verification task. Instead of computing pixel-level overlap, SER uses a referee VLM as a local checker to evaluate model-generated evidence claims across two dimensions: relevance and localization quality, combined with a temporal penalty. This design reduces the reliance on dense box annotations and enables training directly on standard video QA data. On the V-STAR benchmark, SER achieves 49.6% mLGM, improving by 3.0 points over the strong evidence-grounded baseline Open-o3-Video, demonstrating its potential in enhancing both answer accuracy and evidence grounding.
