Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
P$^{3}$-WAM unifies 3D perception, action-conditioned prediction, and goal-directed planning for egocentric dexterous manipulation by integrating a structured 3D action representation into a pretrained video diffusion transformer via a mixture-of-transformers architecture. This unified approach, enabled by randomized modality visibility for flexible conditioning and prediction, significantly improves performance across all three capabilities compared to single-mode training. The model achieves state-of-the-art results in 3D hand reconstruction and action-conditioned video prediction, while also generating more instruction-adherent and progressively complex manipulation trajectories.
A single model now masters 3D hand reconstruction, predicts future video under actions, and plans complex manipulation sequences from language, outperforming specialized methods.
We introduce P$^{3}$-WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal. Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes. Experiments show that unified training improves all three capabilities over single-mode training, including gains in planning quality under a vision-language model evaluation. Through unified training, our model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction adherence and action progression than video-generation baselines. Project Page: https://p3-wam.pages.dev/
ETH