Search papers, labs, and topics across Lattice.
The paper introduces DreamX-Phi 1.0, an action-conditioned video world model designed for robotic manipulation that predicts future observations based on observed frames, language instructions, and action sequences. To enhance realism while ensuring accurate arm movements and object manipulation, the authors incorporate per-arm SE(3) transformations and a depth branch for scene geometry, alongside SAM3 masks for object consistency. Achieving first place in Track 1 and second in Track 2 of the WorldArena 2.0 Challenge, DreamX-Phi 1.0 demonstrates significant advancements in robotic manipulation capabilities.
Realistic predictions in robotic manipulation now come with precise arm control, ensuring the right actions yield the expected outcomes.
We present \textbf{DreamX-Phi 1.0}, an action-conditioned video world model for robotic manipulation that, given an observed frame, a language instruction, and a prescribed action sequence comprising end-effector poses and gripper states, predicts the resulting future observations. Yet realism alone does not guarantee faithfulness: a convincing rollout can still move the wrong arm or lose the manipulated object. To ensure the prediction respects each arm's commanded path, we inject per-arm $\mathrm{SE}(3)$ transformations into attention via \textbf{PRoPE-style geometric encoding}, preserving arm identity and rigid-motion structure. Action control alone does not fully constrain scene geometry or the evolution of small manipulated objects. We therefore add a lightweight \textbf{depth branch} for scene-level geometry and use \textbf{SAM3 masks} with a frozen \textbf{V-JEPA teacher} to maintain object consistency throughout grasping. We further distill the multi-step generator into a few-step student via distribution-matching distillation for efficient deployment. At the time of writing, \model{} achieves first place on Track~1 and second place on Track~2 of the WorldArena~2.0 Challenge. Our model and code will be publicly available.