Search papers, labs, and topics across Lattice.
This paper introduces MobileWAM, a novel mixture-of-transformers architecture that integrates a pretrained video diffusion transformer with a lightweight action expert to enhance mobile manipulation in robotics. By employing a three-expert mixture model that accounts for locomotion and manipulation dynamics, MobileWAM effectively translates internet-scale motion priors into robust whole-body control. The proposed Chain-of-Foresight (CoF) method improves supervision by predicting future latent representations, enabling MobileWAM to outperform existing mobile manipulation policies on benchmark tasks and demonstrating strong generalization on real-world applications.
MobileWAM achieves superior mobile manipulation performance by seamlessly integrating foresight into action planning, outpacing state-of-the-art methods.
World action models (WAMs) built on video generation backbones are a rising recipe for robot learning, yet remain confined to tabletop manipulation. Mobile manipulation demands simultaneous locomotion and whole-body manipulation amid scene-scale dynamics, yet is still dominated by dynamics-blind visual encoders with hand-crafted coordination. We bridge this gap with MobileWAM, a mixture-of-transformers architecture that fuses a pretrained video diffusion transformer with a lightweight action expert through layerwise joint attention, translating internet-scale motion priors into whole-body control. To reconcile the heterogeneous dynamics of moving and manipulating, each feed-forward layer of the action expert becomes a three-expert mixture of shared, locomotion, and manipulation experts, softly routed by the motion intent in the action tokens. To densify supervision, we further propose Chain-of-Foresight (CoF): intermediate representations sequentially predict a chain of future latent chunks, each step conditioned on its predecessor. CoF pairs naturally with our decoupled video--action denoising scheme. At deployment, the WAM serves as a pure current-frame encoder; foresight acts only through gradients, so at inference the foresight chain and video generation are discarded, leaving only policy-level cost. MobileWAM surpasses state-of-the-art mobile manipulation policies on ManiSkill-HAB and fine-tunes to a real ARX Lift2 mobile manipulator across diverse tasks with strong generalization. Code will be released soon.
