Search papers, labs, and topics across Lattice.
This paper introduces Mask2Real-WM, a two-stage action-conditioned world model designed for dexterous manipulation that separates pixel prediction into a dynamics model and a rendering model. By predicting future segmentation masks based on past masks and action sequences, and then translating these masks into photorealistic images, the model effectively reduces the sim-to-real gap, leveraging extensive synthetic data for pretraining. Experimental results demonstrate that the approach significantly enhances per-DoF action controllability, outperforming traditional monolithic models that struggle with fine-grained joint actions.
Segmentation masks can bridge the sim-to-real gap, enabling robots to achieve precise control over 23 degrees of freedom in dexterous manipulation tasks.
Action-conditioned world models allow robots to predict the future consequences of candidate actions without additional physical interaction, supporting policy evaluation, planning, and data augmentation. We present Mask2Real-WM, a two-stage action-conditioned world model for dexterous manipulation that decouples pixel prediction into a dynamics model and a rendering model. The dynamics model predicts future segmentation masks from past masks and 23-DoF action sequences. The rendering model maps the predicted masks to photorealistic RGB using a ControlNet-augmented Stable Video Diffusion backbone. The smaller sim-to-real gap in segmentation space enables the dynamics model to benefit from large-scale pretraining on over 50 h of synthetic simulation data, followed by fine-tuning on fewer than 2.5 h of real demonstrations. Experiments on a dexterous pick-and-place benchmark show that mask conditioning and simulation pretraining are both required for per-DoF action controllability across all 23 degrees of freedom. In contrast, monolithic baselines capture broad hand and end-effector trajectories but do not reliably reflect fine-grained, per-joint action effects.
ETH
Stanford HAI