Search papers, labs, and topics across Lattice.
This paper introduces RoboSeg, a novel part-level semantic reconstruction system for robotic manipulation that integrates vision-language model (VLM) functional-part discovery with asynchronous online RGB-D semantic reconstruction. By utilizing a combination of high-frequency geometry scanning and semantic part mask generation, RoboSeg effectively identifies actionable parts without relying on CAD models or pre-scanned meshes. The system achieves an impressive 83.4% mean part intersection-over-union (mIoU) and demonstrates high task success rates in physical trials, showcasing its potential as a robust semantic indexing layer for task-oriented manipulation.
RoboSeg achieves 83.4% mIoU and flawless grasping in physical trials, redefining how robots can understand and interact with their environment in real-time.
Robotic manipulation requires perception systemsthat identify actionable parts such as handles, rims, triggers,and tool tips, not merely object categories or point clouds. This paper presents RoboSeg, a part-level semantic reconstructionsystem that links vision-language model (VLM) functional-partdiscovery, asynchronous online RGB-D semantic reconstruc-tion, and task-oriented grasp generation without requiring CAD models or pre-scanned meshes. RoboSeg queries a VLM onthe initial RGB observation to obtain compact functional part prompts, then scans with two asynchronous streams: a high-frequency geometry thread for RGB-D odometry and truncated signed distance function (TSDF) fusion, and a keyframe-triggered semantic thread for SAM3 part masks. Projectedmasks are fused by voxel-level temporal voting into a persistentpart-labeled point cloud; RoboSeg uses this map to assign AnyGrasp 6-DoF candidates to semantic parts and select grasps consistent with the task-relevant part label. RoboSeg reaches 83.4% mean part intersection-over-union (mIoU) over manually labeled objects; in a 24-trial physical pilot across fourobjects and eight tasks, the selected grasp contacts the requestedpart in all trials and achieves 21/24 combined task successes.These results characterize RoboSeg as a semantic indexing layerfor task-conditioned manipulation, with AnyGrasp retained asthe proposal generator.
