Search papers, labs, and topics across Lattice.
This paper addresses the challenge of integrating visual scene understanding with autonomous action planning in surgical automation by introducing surgical affordance prediction, which identifies actionable regions for key surgical actions from visual data. The authors propose a novel adaptive feature fusion framework that combines a self-supervised vision transformer encoder for semantic understanding with a generative model encoder for spatial awareness, alongside a hierarchical prompt learning mechanism for context adaptation. Their extensive experiments, validated on a newly established dataset and realistic surgical phantoms, demonstrate that the proposed method achieves state-of-the-art performance and effectively enables autonomous surgical actions.
Surgical affordance maps can now be predicted with unprecedented accuracy, paving the way for true robotic autonomy in complex surgical environments.
Surgical automation is being increasingly studied, yet bridging visual scene understanding with autonomous action planning remains a fundamental challenge. While much research effort has been made on scene perception (e.g., tool recognition and scene segmentation), understanding and predicting actionable possibilities for surgical automation is still underexplored. In this paper, we introduce surgical affordance prediction, which identifies actionable regions for fundamental surgical actions from visual data. Specifically, a novel adaptive feature fusion framework is proposed that leverages the complementary strengths of a self-supervised vision transformer encoder for its superior semantic understanding and a large-scale generative model encoder for its spatially-aware capability. Furthermore, we introduce a hierarchical prompt learning mechanism to adapt to varying procedural contexts. Finally, a scene-guided attention decoder is proposed to focus on critical surgical areas while suppressing background distractions. To validate the effectiveness, we established a new dataset, derived from publicly available surgical datasets with affordance annotations for three basic surgical actions: aspiration, clipping, and retraction. Extensive experiments demonstrate that our approach achieves state-of-the-art performance. Moreover, we validate our framework's applicability for downstream automation on a realistic lung and prostate phantom, and results show that the predicted affordance maps successfully enable autonomous surgical actions.
