Search papers, labs, and topics across Lattice.
This paper introduces a Structured 4D Latent Predictive Model that enhances robot planning by predicting the evolution of a scene's 3D structure based on observations and textual instructions. By encoding scenes in a structured latent space, the model achieves improved 3D consistency and multi-view coherence, outperforming existing 2D video-based planners. Experiments show that this approach not only generates visually high-quality future scenes but also excels in complex manipulation tasks and generalizes well to new visual conditions on real-world robotic platforms.
Achieving superior 3D consistency and multi-view coherence, this model sets a new standard for robot planning in dynamic environments.
Video predictive models are emerging as a powerful paradigm in robotics, offering a promising path toward task generalization, long-horizon planning, and flexible decision-making. However, prevailing approaches often operate on 2D video sequences, inherently lacking the 3D geometric understanding necessary for precise spatial reasoning and physical consistency. We introduce a Structured 4D Latent Predictive Model, which predicts the evolution of a scene's 3D structure in a structured latent space conditioned on observations and textual instructions. Our representation encodes the scene holistically and can be decoded into diverse 3D formats, enabling a more complete and 3D consistent scene understanding. This structured 4D latent predictive model serves as a planner, generating future scenes that are translated into executable actions by a goal-conditioned inverse dynamics module. Experiments demonstrate that our model generates futures with strong visual quality, substantially better 3D consistency and multi-view coherence compared to state-of-the-art video-based planners. Consequently, our full planning pipeline achieves superior performance on complex manipulation tasks, exhibits robust generalization to novel visual conditions, and proves effective on real-world robotic platforms. Our website is available at https://structured-4d-model.github.io/.
