Search papers, labs, and topics across Lattice.
This paper introduces iMac, a novel control paradigm that utilizes raw visual images as action representations in embodied world models, overcoming the limitations of traditional low-dimensional action vectors. By employing a dual-branch architecture that includes an image-action encoder and a dynamic world predictor, iMac achieves superior prediction accuracy and task success rates across diverse robotic scenarios. The findings indicate that this image-based approach not only enhances generalization capabilities but also eliminates the need for predefined action spaces, paving the way for more flexible control in heterogeneous robotic systems.
Treating raw visual images as action representations, iMac significantly boosts prediction accuracy and task success in robotic manipulation, outperforming traditional action vector methods.
Embodied world models have emerged as a pivotal paradigm for visual robotic decision-making and interactive environment simulation. However, conventional embodied frameworks rely on low-dimensional structured action vectors (e.g., joint angles and end-effector poses), which suffer from limited expressive capacity, poor generalization across diverse embodiments, and unnatural dynamic modeling for complex physical interactions. To address these limitations, this paper proposesiMac (Image as Action Control), a novel unified control paradigm that treats raw visual images as native action representations for embodied world models. Departing from traditional explicit kinematic action encoding, iMac formulates continuous visual manipulation as image-based action tokens, which inherently encapsulate spatial motion intentions, interactive geometric constraints and subtle physical dynamics. We construct a dual-branch embodied architecture consisting of an image-action encoder and a dynamic world predictor: the encoder compresses target-driven visual images into compact action embeddings, while the predictor learns environment transition rules conditioned on image actions to achieve high-fidelity future state prediction and closed-loop embodied control. Extensive experiments are conducted on public embodied manipulation benchmarks and real-world robotic scenarios. The results demonstrate that iMac outperforms vector-based action control baselines in prediction accuracy, task success rate and cross-scene generalization ability. Moreover, our image-action design eliminates the reliance on manually defined action spaces, realizing flexible and universal control for heterogeneous embodied agents. This work provides an innovative visual-action perspective for embodied world models, offering a simple yet effective paradigm for scalable robotic perception and manipulation.
Tsinghua AI