Search papers, labs, and topics across Lattice.
STORM introduces a two-stage training framework that internalizes spatial-temporal reasoning within video-language models (LVLMs) using continuous latent trajectories, eliminating the need for explicit textual chain-of-thought or external tools during inference. In the first stage, latent tokens are aligned with thought-video representations derived from generated videos, while the second stage trains the model with answer-only supervision to internalize the reasoning process. Experiments across multiple benchmarks demonstrate that STORM achieves improved video reasoning accuracy with significantly reduced inference overhead compared to methods relying on external tools or video generation.
Ditch the clunky tool-use pipelines: STORM teaches video-language models to reason about space and time using *internalized* latent trajectories, slashing inference costs while boosting accuracy.
Many video reasoning tasks require tracking motion, temporal order, and evolving visual states across frames. Existing methods built on large vision-language models (LVLMs) often address this challenge by externalizing reasoning through textual chain-of-thought (CoT), keyframe selection, repeated frame reinsertion, or external tool use. While effective, such pipelines increase inference-time latency and engineering complexity, and they force temporal-visual evidence to be serialized into text or repeatedly re-encoded from frames. Inspired by the intuition that visual reasoning can occur implicitly before verbalization, we propose STORMS (Spatial-Temporal reasOning via inteRnalized Modeling), a two-stage framework that teaches LVLMs to reason through bounded continuous latent trajectories instead of explicit textual CoT. In Stage I, STORMS aligns latent tokens with thought-video representations derived from generated videos, grounding the latent states in dynamic visual evidence. In Stage II, the model is further trained with answer-only supervision, encouraging the reasoning process to be internalized without step-by-step annotations. Generated thought videos are used only during training; at inference, STORMS performs a bounded latent rollout without regenerating videos, reinserting frames, or invoking external visual tools. Experiments on VideoMME, MVBench, TempCompass, and MMVU show that STORMS improves video reasoning accuracy while substantially reducing inference overhead compared with tool or video-generation-based reasoning pipelines.
NVIDIA