Search papers, labs, and topics across Lattice.
This paper introduces DeVA, a Decoupled Video-Action model designed to enhance robot policy learning by leveraging specialized video and action experts alongside physically salient guidance. By transferring representations from multiple video layers to the action expert and incorporating supervision from affordance and depth information, DeVA effectively improves the adaptability and efficiency of policy learning. Experimental results show that DeVA outperforms traditional unified architectures in both simulation and real-world tasks, achieving faster convergence and better performance with limited data.
DeVA achieves faster convergence and superior performance in robot manipulation tasks by decoupling video and action predictions while integrating physical guidance.
Generalizable robot manipulation requires policies that can anticipate how visual scenes evolve while executing language instructions. While recent Vision-Language-Action models benefit from large-scale pretraining, their predominantly static pretraining objectives provide limited supervision for physical dynamics and temporal causality, leaving control-relevant knowledge to be learned from downstream robot demonstrations. Video generative models offer a promising foundation by encoding rich spatiotemporal priors through future predictions. However, existing Video-Action Models either couple video and action prediction in a shared backbone, making policy adaptation harder to optimize, or under-utilize video information when guiding the action branch. In this work, we introduce DeVA, a Decoupled Video-Action model with specialized video and action experts, multi-level feature transfer, and physically salient guidance. DeVA transfers representations from multiple video layers to the action expert, enabling rich information exchange while making policy learning more tractable. It further supervises intermediate video features and the action stream with physically salient guidance (affordance/depth). Experiments on both simulation benchmarks and real-world deployment demonstrate strong performance with limited data, faster convergence than a unified architecture, and clear performance gains from physical guidance.
