Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the problem of fruit occlusion in agricultural robotics by developing a system that pushes obstructing branches out of the way. They use a deep learning generative model to estimate the occluded part of the fruit in depth space, combined with image processing to determine the push direction. A 3D Hough transform is then applied to point cloud data to detect and select the most relevant branch for the robot arm to push, improving fruit visibility.
Robots can now see the forest *and* the fruit, thanks to a clever combination of deep learning and classical methods that physically clear occlusions.
In agricultural robotics, effective observation and localization of fruits present challenges due to occlusions caused by other parts of the tree, such as branches and leaves. These occlusions can result in false fruit localization or impede the robot from picking the fruit. The objective of this work is to push away branches that block the fruit’s view to increase their visibility. Our setup consists of an RGB-D camera and a robot arm. First, we detect the occluded fruit in the RGB image and estimate its occluded part via a deep learning generative model in the depth space. The direction to push to clear the occlusions is determined using classic image processing techniques. We then introduce a 3D extension of the 2D Hough transform to detect straight line segments in the point cloud. This extension helps detect tree branches and identify the one mainly responsible for the occlusion. Finally, we clear the occlusion by pushing the branch with the robot arm. Our method uses a combination of deep learning for fruit appearance estimation, classic image processing for push direction determination, and 3D Hough transform for branch detection. We validate our perception methods through real data under different lighting conditions and various types of fruits (i.e. apple, lemon, orange), achieving improved visibility and successful occlusion clearance. We demonstrate the practical application of our approach through a real robot branch pushing demonstration.