Search papers, labs, and topics across Lattice.
This paper introduces DynTrace, a novel framework designed to enhance 4D spatio-temporal reasoning in Multimodal Large Language Models (MLLMs) by addressing the limitations of current models in tracking dynamic object evidence. By integrating Dynamic Trajectory Visualization (DTV) and a Dynamic Trace Token (DT-Token) organized into a Dynamic Trace Graph (DTG), DynTrace provides continuous and coherent tracking of dynamic cues, effectively disentangling genuine object dynamics from camera-induced motion. The framework demonstrates significant improvements in performance on multiple benchmarks, establishing a new state-of-the-art in the field and highlighting the critical role of dynamic object evidence in robust 4D reasoning.
Continuous tracking of dynamic object evidence can transform MLLMs' ability to understand and interact with dynamic environments.
4D spatio-temporal reasoning, jointly modeling 3D spatial structure and temporal evolution, is essential for understanding dynamic worlds and enabling embodied interaction. While current Multimodal Large Language Models (MLLMs) show strong capabilities in static scene understanding and coarse-grained 4D tasks, they still have notable limitations in continuous dynamic scene perception, especially in tracking dynamic object evidence for coherent 4D spatio-temporal reasoning. This shortcoming stems mainly from relying on sparse frame-level observations, fragmenting continuous dynamic cues and leaving models unable to disentangle genuine object dynamics from camera-induced apparent motion. Inspired by humans tracking dynamic cues while compensating for viewpoint changes, we propose DynTrace, a training-free framework for 4D spatio-temporal reasoning with two complementary components. Dynamic Trajectory Visualization (DTV) reprojects world-coordinate trajectories onto the image plane, providing geometry-informed visual priors that disentangle genuine object dynamics from camera-induced apparent motion. Meanwhile, the Dynamic Trace Token (DT-Token), organized into a Dynamic Trace Graph (DTG), tracks object-level dynamic cues, trace evolution, and key moments, maintaining continuous dynamic object evidence for coherent 4D reasoning. Together, these two components equip MLLMs with continuously tracked dynamic object evidence, grounded in geometry-informed visual priors and structured spatio-temporal traces. DynTrace consistently improves open-source MLLMs, achieving state-of-the-art results on Dyn-Bench, VLM4D, and DSI-Bench, validating the importance of tracking dynamic object evidence for robust 4D spatio-temporal reasoning.
Tsinghua AI