Search papers, labs, and topics across Lattice.
This paper introduces Latent Dynamics Reasoning (LDR), a novel approach that explicitly models the kinematic integration of latent transitions to capture the dynamics of video data. By focusing on structured latent representations rather than dense convolutional features, LDR significantly improves extrapolation of learned dynamics, achieving over 20 times smaller error gaps in out-of-distribution scenarios compared to existing video diffusion models. Notably, LDR operates with 26 times fewer parameters and is 143 times faster, demonstrating its efficiency and effectiveness in generalizing to unseen dynamics.
LDR achieves unprecedented extrapolation of learned dynamics in video models, outperforming traditional methods by a staggering margin in both accuracy and efficiency.
The world evolves following its dynamics, i.e., its laws of motion. However, leading video diffusion models largely fit the pixels without modeling how the pixels transit over time. Thus, they render visually plausible frames but may not accurately obey the laws. To capture the dynamics purely from pixels, we introduce Latent Dynamics Reasoning (LDR). LDR casts the latent transition as an explicit kinematic integration, where the lower-order dynamics are integrated numerically and the model regresses only the third- and higher-order residual that drives the rollout. For this integration to extrapolate better, LDR runs it on a structured latent rather than dense convolutional features. Following PhyWorld, we validate LDR on a controlled white-box physics benchmark spanning five tasks (uniform motion, parabola, collision, bouncing, looming), focusing on out-of-distribution scenarios that reveal whether a model has truly learned the underlying dynamics. LDR extrapolates the learned dynamics far better: the gap between its in- and out-of-distribution error is over 20$\times$ smaller than the video diffusion baseline's, under both single- and joint-task training at 256$^2$ resolution, while using 26$\times$ fewer parameters and running 143$\times$ faster. LDR can even generalize under severe shift: for example, trained only on red balls moving left-to-right, it correctly predicts the motion of a blue square moving right-to-left. To our knowledge, this is the first video world model that extrapolates learned dynamics beyond its training distribution. Project page: https://lat-dyn-reason.github.io/
