Search papers, labs, and topics across Lattice.
This paper introduces HarvestPoint-ACT, an end-to-end imitation learning framework that explicitly addresses the challenges of target selection and harvest-point conditioning in robotic fruit harvesting, particularly under occlusion. By employing an instance segmentation front end with a keypoint branch, the system predicts both a mask and a harvest point for each visible fruit, while a scheduler ranks candidates based on occlusion and travel distance. The method achieves a notable success rate of 88% in a mock-up environment, demonstrating robust performance even with a 75% success rate under heavy occlusion conditions.
HarvestPoint-ACT achieves an impressive 88% success rate in robotic fruit harvesting, even under challenging occlusion scenarios.
End-to-end imitation learning avoids hand-made robot motion for approaching and grasping, but the policy must still decide which fruit to pick and where to close the gripper. Occlusion can make the policy lose the selected fruit during harvesting, and the correct closing point is difficult to infer from pixels alone. This paper presents HarvestPoint-ACT, which makes both decisions explicit in perception and provides them to the policy. An instance segmentation front end with a keypoint branch predicts a mask and a harvest point for each visible fruit, where the harvest point specifies the location to close the gripper. A scheduler ranks detected candidates by occlusion and travel distance and selects one target. After each attempt, it redetects and reranks the candidates because the canopy may have changed. The selected fruit is encoded for an action chunking transformer as an eight-dimensional state, containing the absolute harvest point, the vector from the gripper to that point, a validity flag, and a confidence score. When the selected fruit is temporarily undetected, the system retains the last harvest point estimate in the robot base frame and marks it as stale, and aborts the attempt if the loss persists. On a canopy mock-up, HarvestPoint-ACT achieves a success rate of 88%, and of 75% under heavy occlusion.
