Search papers, labs, and topics across Lattice.
This paper introduces the Hierarchical Prototype-Memory Adaptation (HPMA) framework for enhancing the Segment Anything Model (SAM) in surgical instrument segmentation (SIS). By constructing a multi-scale visual prototype memory bank and integrating it into SAM's feature space, HPMA overcomes the limitations of traditional prompt-learning methods that degrade model robustness in complex surgical environments. The approach achieves state-of-the-art performance on the EndoVis2017 and EndoVis2018 datasets, significantly improving segmentation accuracy under challenging conditions.
Adapting SAM with a hierarchical memory framework boosts surgical instrument segmentation performance beyond existing methods, revealing the critical role of stable category memory in dynamic environments.
Surgical instrument segmentation (SIS) is fundamental for computer-assisted surgery, where reliable instrument masks enable precise scene understanding and clinical assistance. Recently, adapting foundation models like the Segment Anything Model (SAM) to the surgical domain via prompt-learning has shown encouraging results. However, the performance of these adapted models under challenging surgical conditions is constrained by suboptimal adaptation mechanisms. Specifically, optimizing prompts or prototypes purely via downstream segmentation loss tends to cause them to degenerate into task-specific parameters rather than serving as persistent, stable category memory, thereby degrading their robustness against complex intraoperative variations. Moreover, routing multi-scale visual cues through a single prompt pathway creates a bottleneck that hinders effective scale-matched coupling. To address these limitations, we propose HPMA, a Hierarchical Prototype-Memory Adaptation framework for SAM. Specifically, HPMA constructs a frozen, multi-scale visual prototype memory bank from annotated surgical scenes and integrates it into SAM's feature space using lightweight adapters to preserve stable category evidence. To maximize the utility of multi-scale cues, we introduce a scale-matched coupling mechanism where global prototypes calibrate class-level prompt features, structural prototypes guide decoder object queries, and local prototypes align high-resolution feature maps through a local alignment objective. Extensive experiments on the public EndoVis2017 and EndoVis2018 datasets demonstrate that our approach achieves state-of-the-art performance, outperforming existing foundation model adaptation methods.
