Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
FutureDuet addresses the divergent visual dynamics in World Action Models by decoupling the future supervision applied to static scene cameras versus dynamic wrist cameras. While maintaining both camera feeds for control via an ActionDiT policy, the framework applies mask- and skeleton-guided future RGB synthesis to the main view and compact future latent prediction to the wrist view, keeping all auxiliary predictors training-only to avoid inference overhead. This asymmetric supervision achieves 94.2% clean and 94.1% randomized success on RoboTwin50 alongside 99.2% on LIBERO, outperforming Fast-WAM by up to 12.8% on contact-rich manipulation tasks.
Forcing moving wrist cameras to predict future pixels actively degrades policy performance—decoupling static scene reconstruction from dynamic wrist latent prediction yields up to a 12.8% boost in contact-rich manipulation with zero inference cost.
World Action Models (WAMs) augment robot action generation with future visual supervision. Existing WAMs commonly fuse main and wrist observations into one visual stream and train both with the same future-video objective, despite their different visual dynamics. A stable main camera reveals scene-level task evolution, whereas wrist cameras move with the end effector, mixing local interaction changes with viewpoint shifts and self-occlusion. These contrasting predictive demands suggest that the two views may benefit from different future objectives. We introduce FutureDuet, which retains both views for control, while allowing each visual stream to receive a different future objective. For the main view, future RGB models task evolution, while interaction masks and robot skeletons focus supervision on task objects and robot motion. For the wrist stream, future latent prediction models short-horizon interaction changes without requiring pixel-level reconstruction. ActionDiT jointly reads the resulting Task State and Interaction State, combining scene-level progress with close-range interaction evidence. All auxiliary prediction modules are training-only, adding no inference overhead. FutureDuet achieves 94.2% clean and 94.1% randomized success on RoboTwin50 and 99.2% average success on LIBERO. The improvements are most pronounced on six RoboTwin50 tasks that require precise interaction, averaging gains of 9.2% and 12.8% over Fast-WAM in clean and randomized settings. Controlled studies further show complementary gains from separating the wrist pathway and designing future supervision separately for the two views.
Tsinghua AI