Search papers, labs, and topics across Lattice.
OmniReasoner addresses the challenges of long audio-video reasoning in omnimodal LLMs by introducing a tool-use framework that enables models to selectively zoom in on relevant segments for higher fidelity analysis. Utilizing a combination of supervised fine-tuning and reinforcement learning, the model first generates a low-cost global preview before determining the need for a detailed inspection. Key results demonstrate that OmniReasoner significantly enhances answer accuracy and temporal grounding while optimizing computational resources by focusing on informative regions of the input data.
By intelligently deciding when to zoom in on audio-video content, OmniReasoner boosts reasoning accuracy while reducing computational costs, transforming how LLMs handle complex multimodal inputs.
Long audio-video reasoning is difficult for omnimodal LLMs because the decisive evidence is often sparse, cross-modal, and too expensive to preserve with uniformly high-fidelity inputs. We introduce OmniReasoner, a tool-use post-training framework for Thinking with Long Audio-Video: omni-modal LLMs learn, via supervised fine-tuning and reinforcement learning, to decide whether and where to call a zoom-in tool before answering. OmniReasoner first builds a low-cost global preview of the full stream and then, when needed, calls the zoom-in tool with a requested temporal interval for higher-fidelity visual and audio inspection before answering. Because the model observes different sampling granularities before and after this call -- a sparse global preview and a denser local clip -- we introduce TimeAnchor, which keeps the tool's temporal argument valid and round-trip-consistent across these granularities, rather than tied to frame indices from a particular sampling rate. To make this tool-use behavior trainable without expensive manual interval annotation, we build a Temporal Augmented Data Engine that synthesizes tool-use post-training trajectories by video editing and composition. Experiments across omnimodal and video benchmarks show that OmniReasoner improves both answer accuracy and temporal grounding while concentrating high-fidelity computation on informative regions. Code is available at https://github.com/RockyChen0205/OmniReasoner.
Tsinghua AI