Search papers, labs, and topics across Lattice.
This paper introduces a hybrid correspondence-based representation for robotic manipulation of rigid-deformable interactions, addressing the limitations of traditional object-centric representations in complex tasks like dressing or handling clothes. By integrating structure-, task-, and interaction-aware sparse keypoints with dense correspondences, the method captures intricate interaction dynamics while ensuring accurate tracking of deformable objects. Experimental results show that this approach allows for effective one-shot transfer to new tasks, significantly enhancing the capabilities of household robots in real-world scenarios.
Sparse keypoints combined with dense correspondences enable robots to master complex rigid-deformable interactions with minimal demonstrations.
Manipulation involving rigid-deformable interactions, such as hanging clothes or dressing humans, is common in daily life, making it essential for household robots. Compared to single-object manipulation or interactions between rigid bodies, these tasks are particularly challenging due to the rich multi-point contacts and the complex dynamics of the deformable bodies during interaction. Therefore, object-centric representations such as 6D poses or structural points without task-specific information become insufficient for these interactions. In this work, we propose a hybrid correspondence-based representation tailored for rigid-deformable interactions. First, to capture intricate interaction information, we introduce structure-, task-, and interaction-aware sparse keypoints. The keypoints are generated based on the global structures of both rigid and deformable objects, and filtered by their local interaction contacts. However, tracking these sparse keypoints through the interaction remains difficult due to the high-dimensional dynamics of deformable objects. Therefore, we further construct dense correspondences on the deformable objects for accurate keypoint tracking throughout the manipulation. This hybrid design combines the advantages of both representations: sparse keypoints encode rich, task-specific information for fine-grained manipulation, while dense correspondences ensure efficient tracking and generalization to novel deformations, shapes, and scenarios. Together, they enable one-shot transfer to new tasks with minimal demonstrations. Extensive experiments demonstrate the effectiveness and broad applicability of our method.
