Search papers, labs, and topics across Lattice.
This paper introduces 3D Point World Models (3DPWM), a novel approach that enhances robotic control by completing partial point clouds to create a fully realized 3D environment before learning action-conditioned dynamics. The method addresses the limitations of existing video-based dynamics models, which struggle with geometric consistency and prediction drift, by ensuring reliable long-horizon rollouts and accurate cost evaluations. Experimental results show that 3DPWM significantly improves performance across various robotic tasks, achieving reliable rollouts of over 100-300 steps and successful sim-to-real transfer.
Reliable long-horizon rollouts in robotic control are now achievable with a 3D model that completes point clouds, overcoming the pitfalls of traditional video-based dynamics.
Learning predictive models of the world enables robotic control through planning, potentially allowing robots to improvise solutions on new tasks. However, large video-based dynamics models lack explicit 3D spatial structure and suffer from geometrically inconsistent long-term rollouts with compounding errors. Emerging 3D dynamics models based on partial point clouds improve geometric consistency but remain sensitive to occlusions and accumulated prediction drift. To address these challenges, we present 3D Point World Models (3DPWM) - a task-agnostic world model that operates entirely in 3D space by first completing partial point clouds and then learning action-conditioned dynamics in this completed 3D scene. By operating on completed geometry, 3DPWM enables reliable long-horizon rollouts and more accurate cost evaluation for model-based planning while supporting adaptation to new tasks. Experiments across different robotic embodiments and tabletop manipulation benchmarks demonstrate that 3DPWM achieves significantly more reliable long-horizon rollouts (100-300+ steps), supports both open-loop and closed-loop planning, and enables successful sim-to-real transfer.
