Search papers, labs, and topics across Lattice.
This paper introduces JEPA-WAM, a novel approach that enhances robot manipulation by integrating a stage-level semantic future with a short-term physical future, allowing robots to better understand task progression. By employing a goal-conditioned Joint-Embedding Predictive Architecture (JEPA) alongside a Motus-based World Action Model (WAM), the system effectively predicts the next stage of a task based on current observations and instructions. The results demonstrate a significant improvement in task success rates and efficiency, achieving a 90.25% success rate across 50 RoboTwin 2.0 tasks while reducing execution steps by 5.97% compared to the best existing methods.
Robots can now predict not just immediate actions but also the next stages of complex tasks, leading to more efficient manipulation strategies.
Generalist robot policies aim to map multimodal observations and linguistic task instructions to actions across diverse tasks. However, existing methods typically represent the future as a fixed, short video-action chunk. This short-term future captures local scene evolution for action execution, but it does not explicitly describe the stage-level future that specifies how a task should progress from its current stage to the next. We therefore distinguish two complementary futures for robot manipulation: a short-term physical future to capture local scene evolution and a stage-level semantic future to represent task progress. We introduce JEPA-WAM, which augments a Motus-based World Action Model (WAM) with Stage-JEPA, a goal-conditioned Joint-Embedding Predictive Architecture (JEPA) predictor. Given the current observation and task instruction, Stage-JEPA uses a frozen V-JEPA2 encoder to extract the current-state representation and predicts the latent target of the next inferred stage. Across 50 RoboTwin 2.0 tasks in clean and randomized environments, JEPA-WAM achieves 90.25% overall success and reduces the mean number of execution steps in successful rollouts by 5.97% relative to the strongest baseline.
Tsinghua AI