Search papers, labs, and topics across Lattice.
This paper introduces video-SALMONN-R$^3$, an innovative end-to-end video large language model that enhances video understanding by employing a two-stage paradigm for efficient question answering. By utilizing reinforcement learning to facilitate a re-watch mechanism without the need for chain-of-thought cold-starts, the model avoids costly data annotations and retains its pretrained capabilities. Experimental results demonstrate that video-SALMONN-R$^3$ not only outperforms baseline models but also achieves superior performance with reduced computational costs compared to previous re-watch methods.
Reinforcement learning enables video-LLMs to re-watch and refine answers without the costly overhead of chain-of-thought training, achieving better performance with less computation.
Video large language models (LLMs) are often constrained by computation and memory budgets, leading them to use reduced frame rates and spatial resolutions, which may cause them to miss critical information for question answering (QA). A practical and efficient solution is a two-stage paradigm: first perform coarse video understanding to localize relevant segments, and then re-watch these segments at higher temporal or spatial fidelity. In this paper, we present video-SALMONN-R$^3$, the first end-to-end video-LLM that enables re-watch through reinforcement learning without relying on chain-of-thought (CoT) cold-start. This design removes the need for costly CoT data annotations and avoids CoT-based supervised fine-tuning (SFT), which can otherwise degrade the pretrained video understanding abilities. To address the mismatch between the reasoning-first behavior induced by re-watch and the answer-first tendency of pretrained video-LLMs, we propose a re-answer strategy, in which the model first produces a direct answer in the first watch and then refines it after re-watching. Finally, to improve question adherence during re-watching, we propose a re-ask mechanism that re-injects the query when revisiting localized segments. Experimental results show that video-SALMONN-R$^3$ consistently outperforms both the base model and the QA-SFT baseline, while surpassing prior re-watch-based approaches with significantly lower computational cost. Code, models, and data will be publicly released upon acceptance.
Tsinghua AI