Search papers, labs, and topics across Lattice.
This paper explores action-conditioned world modeling to develop transferable dynamics priors for robot learning, leveraging a multi-view interactive base diffusion model called A2World pretrained on extensive robot manipulation data. The study validates these dynamics priors through two applications: creating a specialized simulator, A2World-sim, for policy evaluation and what-if analysis, and adapting A2World into A2World-policy for predicting actions based on visual and instructional inputs. Results indicate that the pretrained model significantly enhances both simulator-centric and policy-centric robot learning, showcasing its versatility and effectiveness in real-world scenarios.
Action-conditioned world modeling can yield reusable dynamics priors that enhance robot learning across both simulation and real-world applications.
We study action-conditioned world modeling as a scalable way to learn transferable dynamics priors for robot learning. By pretraining a model to predict how actions drive visual scene evolution, the resulting world model captures reusable interaction dynamics beyond appearance-level video generation. Concretely, we pretrain a multi-view interactive base diffusion world model, A2World, on large-scale robot manipulation data with real action annotations. We validate the learned dynamics priors from two complementary perspectives. First, we adapt A2World into a task- or scene-specialized real-world simulator, A2World-sim, whose long-horizon rollouts support simulator-based policy evaluation and scalable what-if analysis by replacing real-robot rollouts with world model rollouts. Second, starting from the same pretrained weights, we adapt A2World into a video-action joint prediction model, A2World-policy, that predicts actions under visual and instruction conditioning. Experiments across simulation benchmarks and real-robot settings demonstrate that action-conditioned world model pretraining yields transferable dynamics priors that benefit both simulator-centric and policy-centric robot learning.
