Search papers, labs, and topics across Lattice.
This paper introduces VICX, a novel framework for robot manipulation that integrates video generation with an In-Context Operator Network to enable generalizable execution across diverse tasks and robot embodiments. By utilizing a frozen video generation model for high-level visual planning and a task-agnostic interface for grounding these plans into executable trajectories, VICX achieves robust performance in unseen scenarios. Experimental results on Meta-World highlight its capabilities in cross-task generalization, closed-loop self-correction, and adaptability to different robotic platforms, showcasing significant advancements in robotic manipulation.
VICX achieves dual generalization in robot manipulation, enabling effective execution across diverse tasks and embodiments without the need for parameter updates.
Generalizable robot manipulation requires not only task-level reasoning over unseen scenes, but also reliable grounding of visual plans into embodiment-specific execution. To bridge this gap, we propose VICX (Video generation and In-Context eXecution), a decoupled closed-loop manipulation framework. In VICX, a frozen video generation model produces vision-language-conditioned high-level visual plans, while a Video-to-Trajectory In-Context Operator Network (V2T-ICON) serves as the task-agnostic interface that grounds these plans into executable robot-state trajectories. To improve execution generalization, V2T-ICON operates on segmentation-extracted arm-only frame observations and uses retrieved image-state pairs as in-context prompts, allowing a robust and generalizable visual-to-state mapping at inference time without parameter updates. Experiments on Meta-World show that VICX supports cross-task generalization, closed-loop self-correction, and cross-embodiment transfer, demonstrating dual generalization across both task semantics and robot execution. The project webpage can be found here: https://scaling-group.github.io/vicx/.
