Search papers, labs, and topics across Lattice.
The Unified Motion-Action (UMA) Model integrates 3D object motion trajectories to unify visuomotor control and dynamics modeling, treating object motion and robot actions as interdependent variables. By employing a masked generative objective, UMA facilitates multi-task pretraining across diverse data sources without the need for manual task annotations, leveraging hindsight-relabeled motion contexts and a contrastive objective to separate task intent from scene geometry. The model demonstrates superior performance compared to specialized state-of-the-art baselines in various tasks, showcasing its versatility and efficiency in real-world applications.
UMA achieves remarkable multi-task performance by seamlessly linking object motion and robot actions, outperforming specialized models without manual task instructions.
We present Unified Motion-Action (UMA) Model, an approach that uses 3D object motion trajectories as a shared interface to bridge visuomotor control and dynamics modeling. UMA treats object motion and robot actions as co-evolving variables under a masked generative objective, in which the mask pattern determines both the supervision regime during pretraining and the inference mode at deployment. Using hindsight-relabeled motion contexts and a contrastive objective that disentangles task intent from scene geometry, UMA enables multi-task pretraining across heterogeneous data sources without requiring manually annotated task instructions. At deployment, the same pretrained parameters support motion-conditioned visuomotor control, motion-based dynamics modeling, and task adaptation from few-shot demonstrations. Pretrained on a mixture of robot demonstrations, human videos, and simulated data, UMA consistently outperforms state-of-the-art baselines specialized for each inference mode.
