Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Multi Rigid Object Gaussian World Model (MRO-GWM), which learns action-conditional dynamics for rigid objects in 3D environments by utilizing object-centric Gaussian representations. The model employs a novel spatio-temporal transformer architecture to predict future motions based on historical object data and planned actions, effectively handling occlusions and partial observations. Evaluation on synthetic datasets demonstrates superior prediction performance and successful application in model-predictive control for non-prehensile manipulation tasks.
Object-centric Gaussian representations enable precise predictions of rigid body dynamics, even in cluttered environments with occlusions.
World models enable intelligent agents to predict the consequences of their actions on the environment. In this paper, we propose Multi Rigid Object Gaussian World Model (MRO-GWM), a novel model that learns action-conditional dynamics of rigid objects in 3D. By representing the scene by object-centric Gaussians, we can represent arbitrary object shapes and multi-object scenes. We develop a novel spatio-temporal transformer architecture that predicts future rigid body motion from a history of object Gaussians and future actions. Objects are represented by their Gaussians in a canonical frame, which allows for describing object motion as rigid body transformation. Our model is trained on reconstructions from multiple viewpoints, which requires the model to handle partial observations of objects due to occlusions. We analyze prediction performance of our approach on synthetic datasets composed of typical household objects with multi-object dynamics and interactions by a robot end effector. We also evaluate our model in model-predictive control for non-prehensile manipulation in simulation.