Search papers, labs, and topics across Lattice.
This paper investigates whether modular grasp synthesis pipelines, which first estimate object pose and shape using recent encoder-decoder and diffusion models and then apply antipodal grasp sampling, can outperform end-to-end grasp synthesis methods. Through experiments with parallel jaw grippers, 7-DoF grasps, and single-view RGB(-D) images, the authors find that modular methods consistently outperform end-to-end approaches, particularly for small objects. However, the performance of modular methods is limited by the accuracy of the underlying pose and shape estimation, especially in cluttered scenes.
Modular grasp synthesis pipelines that decompose grasping into pose/shape estimation followed by antipodal grasp sampling can surprisingly outperform end-to-end methods, even leveraging vision-language models for language-conditioned grasping.
The problem of object pose and shape estimation has seen key advancements lately. Encoder-decoder (e.g., SAM3D, LRM, CRISP) and diffusion-based models (e.g., InstantMesh, Zero123, SceneComplete) have shown category-agnostic shape encoding capacity and open-set generalizability. In this work, we ask the question: Are the object pose and shape estimation methods mature enough, such that when used with antipodal grasp sampling, can outperform the end-to-end grasp synthesis methods? We explore this question in detail by scoping our study to parallel jaw grippers, 7-DoF grasps, and single-view RGB(-D) image as input. We implement and compare a state-of-the-art, end-to-end grasp synthesis method and three modular methods, which first estimate the object pose and shape for all objects in the scene, and generate grasps using antipodal sampling. We observe that the modular methods outperform the end-to-end method in all our experiments. The modular methods are able to synthesize plenty of grasps, even for small objects, where the end-to-end methods fail. The effectiveness of the modular methods is contingent on the accuracy of the pose and shape estimation, and suffers partial degradation in cluttered scenes - a limitation of the existing pose and shape estimation methods. We also analyze the failure modes and run-times for the three modular methods, which use two different ways of object pose and shape estimation: one based on an encoder-decoder model, while another a diffusion model. Finally, we demonstrate that the single-view object pose and shape estimation methods can be augmented with vision-language models to yield language-conditioned grasps from just single-view RGB-D image as input. We notice comparable performance to the state-of-the-art LERF-TOGO baseline.
