Search papers, labs, and topics across Lattice.
The paper introduces StateTrace, an object-centric framework designed to enhance hidden-state spatiotemporal reasoning in long videos, addressing the limitations of existing video language models (VLMs) that misinterpret invisible objects as unknown. By constructing a reusable spatiotemporal state memory that organizes object trajectories and state transitions, StateTrace allows VideoLLMs to effectively infer object states during periods of invisibility. Experimental results demonstrate significant performance improvements, such as a boost from 39.6 to 64.2 on the newly proposed HSR-Bench, showcasing the framework's efficacy in long-video reasoning tasks.
StateTrace reveals that enhancing memory structures in VLMs can drastically improve reasoning about invisible objects, achieving a remarkable 24.6-point performance increase on hidden-state tasks.
Existing VLMs have achieved strong performance in video understanding, yet they struggle with long-video spatiotemporal reasoning when target objects become invisible, often mistaking"invisible"for"unknown". We define this challenge as hidden-state spatiotemporal reasoning: inferring object states during prolonged invisible intervals from context interactions. To address this, we propose StateTrace, a novel object-centric framework that endows VideoLLMs with an explicit mechanism for hidden state reasoning in long videos. StateTrace builds a reusable spatiotemporal state memory that organizes object trajectories, inter-object relations, and state-transition events into a structured reasoning substrate. At inference time, it retrieves question-relevant state-evolution trajectories and converts them into compact reasoning cues, enabling the model to explicitly reason about why an object disappears, how its state evolves while invisible, and whether that state should persist at query time. We further build HSR-Bench, a diagnostic benchmark for hidden-state reasoning, containing 1,427 video-QA samples from 1,384 unique videos. Extensive experiments across multiple VideoLLMs show that StateTrace consistently improves performance on both public benchmarks and HSR-Bench (e.g., improving VideoLLaMA3 from 39.6 to 64.2 on HSR-Bench).
