Search papers, labs, and topics across Lattice.
This paper introduces OmniAgent, a novel omni-modal agent that reformulates video understanding through a POMDP-based iterative Observation-Thought-Action cycle, allowing it to selectively process audio-visual cues and maintain a persistent textual memory. By employing Agentic Supervised Fine-Tuning and Agentic Reinforcement Learning with TAURA, OmniAgent effectively decouples reasoning complexity from video length, leading to improved performance as the number of reasoning turns increases. Empirical evaluations reveal that OmniAgent achieves state-of-the-art results across ten benchmarks, outperforming significantly larger models, such as Qwen2.5-VL-72B, on key tasks.
OmniAgent's ability to improve performance with more reasoning turns challenges the traditional "watch-it-all" approach in video understanding.
Passive models for long video understanding typically rely on a "watch-it-all" paradigm, processing frames uniformly regardless of query difficulty, causing computational cost to grow with video duration. Although interactive frameworks have emerged, they often rely on global pre-scanning, and their context cost still scales with video length. We propose OmniAgent, the first native omni-modal agent that formulates video understanding as a POMDP-based iterative Observation-Thought-Action cycle. OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory, effectively decoupling reasoning complexity from raw video duration. To operationalize this, we introduce (1) Agentic Supervised Fine-Tuning to bootstrap native active perception via best-of-N trajectory synthesis with dual-stage quality control, and (2) Agentic Reinforcement Learning with TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage), which leverages turn-level entropy to steer credit assignment toward pivotal discovery turns. Crucially, OmniAgent exhibits positive test-time scaling, where performance improves as the number of reasoning turns increases, validating the efficacy of active perception. Empirical results across ten benchmarks (e.g., VideoMME, LVBench) demonstrate that OmniAgent achieves state-of-the-art performance among open-source models. Notably, on LVBench, our 7B agent outperforms the 10times larger Qwen2.5-VL-72B (50.5% vs. 47.3%).
DAMO