Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
World Motion Models (WMMs) represent dynamic 3D scenes using sparse SE(3) pose trajectories, unifying diverse elements like articulated objects and human motion into a single generative prior. This flexible sequence modeling approach, implemented via flow-matching with per-token noise and a context token mechanism, enables arbitrary marginal conditioning across entities and time. WMMs demonstrate versatility by reducing tasks from future prediction to policy learning to masked inference on a single network, achieving strong performance across six diverse applications.
A unified generative model for 3D dynamics can predict, inpaint, and control complex SE(3) trajectories across diverse agents and scenes with a single network.
Equipping artificial agents with spatial intelligence requires a comprehensive generative prior over the dynamic 3D world. We propose World Motion Models (WMMs) that capture "what was, is, and will be where across time" via sparse SE(3) pose trajectories. WMMs are built on the observation that elements of dynamic scenes can be well approximated by a set of rigid SE(3) trajectories, a minimal yet expressive primitive for 4D modeling. This representation unifies articulated objects, human bodies, hand-object interactions, piecewise-rigid scene dynamics, camera motion, and even robot states and actions into a single shared space. Given this representation, we cast the joint distribution of these entities as a flexible sequence modeling problem, utilizing flow-matching with per-token noise levels. Coupled with a context token mechanism for non-sequential conditioning, this formulation supports any-to-any marginal conditioning across an arbitrary number of entities and time steps. Tasks such as future prediction, motion infilling, model-predictive control, inverse kinematics, cross-embodiment retargeting, and policy learning all reduce to the application of different masks over the same network. Experiments on 6 diverse applications of 3D vision and robotics demonstrate the versatility and flexibility of WMMs with strong performance.
BAIR