Search papers, labs, and topics across Lattice.
This paper introduces VISTA-Policy, an innovative imitation learning framework that leverages a Visual Deformation Field (VDF) to enhance contact-rich manipulation tasks by providing high-dimensional visuo-physical feedback. By integrating a Physics-Aware Encoding Engine, an Energy Aggregation Denoising Mechanism, and a Deformation-Augmented Policy Network, the method achieves superior performance in tasks like object grasping and calligraphy writing compared to both vision-only and tactile-based approaches. The results highlight VISTA-Policy's exceptional out-of-distribution generalization and robustness, making it a promising solution for fine-grained manipulation in complex environments.
VISTA-Policy outperforms traditional vision and tactile methods, achieving remarkable robustness and generalization in contact-rich manipulation tasks.
Contact-rich manipulation requires precise interaction feedback. While vision-centric imitation learning is prevalent, external visual observations provide indirect and ambiguous cues about contact states, particularly under occlusion or subtle object--gripper interactions; dedicated tactile or force sensors can provide rich contact information but introduce additional hardware complexity, calibration requirements, and deployment costs. To bridge this gap, we propose VISTA-Policy, an imitation learning paradigm that utilizes the Visual Deformation Field (VDF), a 3D displacement representation of a compliant gripper, as high-dimensional visuo-physical feedback. The framework integrates: 1) a Physics-Aware Encoding Engine for real-time VDF decoding; 2) an Energy Aggregation Denoising Mechanism to isolate true interaction signals; and 3) a Deformation-Augmented Policy Network with incremental gripper actions for precise closed-loop correction. Extensive evaluations on Cross-Scale Object Grasping, Cap Unscrewing, and Calligraphy Writing demonstrate that VISTA-Policy outperforms the strong pure-vision baseline 3D Diffusion Policy and the tactile baseline. VISTA-Policy further demonstrates substantial out-of-distribution generalization to unseen object scales and robustness against dynamic disturbances, offering a durable and cost-effective route toward general-purpose fine-grained manipulation in unstructured environments. Project videos and supplementary materials are available at: https://sites.google.com/view/vista-policy.
