Search papers, labs, and topics across Lattice.
This paper introduces a monocular vision-based grasping framework that effectively handles both soft and rigid objects using only RGB input and a position-controlled gripper. By integrating open-vocabulary object detection, real-time point tracking, and a language-based stiffness estimation model, the system adapts its grasping strategy based on the mechanical properties of the objects. Experimental validation shows that the framework achieves stable grasping across diverse materials, making it a practical solution for unstructured environments without reliance on tactile sensing or specialized equipment.
Achieving stable grasping of both soft and rigid objects using only visual input could revolutionize robotic manipulation in everyday settings.
Grasping in unstructured environments requires handling objects with widely different mechanical properties, from soft and deformable items to rigid everyday objects. Most existing approaches address these categories separately and often rely on tactile sensing, object-specific models, or specialized grippers. In this paper, we present a unified monocular vision-based grasping framework that targets both soft and rigid objects within a single control pipeline, using only RGB input and a position-controlled gripper. The proposed system combines open-vocabulary object detection, image segmentation, boundary-aware point assignment, real-time point tracking, and monocular depth estimation to recover object motion and geometry from visual observations. A key component of the framework is a language-based stiffness estimation model that infers an object's expected compliance from its semantic description and provides an object-level prior for selecting the grasping strategy before contact. For deformable objects, grasp adaptation is governed by a Procrustes-based dissimilarity measure computed from tracked keypoints, which acts as a visual proxy for deformation. For rigid objects, the gripper width is regulated through the scaling of tracked point distances. We validate the proposed method in real-world pick-and-place experiments on a Franka Emika Research 3 arm using objects with substantially different mechanical properties, including lettuce, fresh mozzarella cheese, croissants, paper towels, and hard plastic bottles. Results demonstrate that the framework achieves stable grasping across both soft and rigid objects using visual feedback alone, highlighting a practical, sensor-efficient, and generalizable approach for food handling and household manipulation.
