Search papers, labs, and topics across Lattice.
This study introduces VLM-hyster, the first vision-language model designed for pixel-wise segmentation of hysteroscopic surgical scenes, addressing challenges such as morphological similarities among lesions and artifacts in surgical videos. By leveraging a pretrained image encoder and a transformer-based decoder, the model enhances segmentation accuracy through category-specific text prompts and a masked distillation branch that filters irrelevant visual features. Experimental results show that VLM-hyster significantly outperforms existing state-of-the-art models and demonstrates robustness and generalizability validated by gynecologists across multiple centers.
VLM-hyster achieves unprecedented segmentation accuracy in hysteroscopic surgeries, outperforming all existing models and paving the way for more effective AI-assisted surgical interventions.
Hysteroscopic surgical scene segmentation plays a pivotal role in understanding the hysteroscopic intraoperative environment as well as computer-assisted intervention. However, this task presents unique challenges due to the high morphological similarity among different lesions and the presence of artifacts such as specular reflections, motion blur, and fluid occlusions in surgical videos. In this work, we propose the first vision-language model (VLM)-based hysteroscopic surgical scene segmentation method, which performs pixel-wise localization for fifteen representative categories in hysteroscopic surgical scenes. Our VLM-hyster has a segmentation backbone that utilizes the pretrained image encoder for robust visual feature extraction, coupled with a transformer-based decoder for dense prediction. Moreover, we design category-specific text prompts and incorporate a masked distillation branch to filter out visual features with low correlation to the text prompts, enabling the model to focus more effectively on category-specific image regions and thereby enhancing segmentation performance. We collect a large multicentric hysteroscopic surgical scene dataset, containing 4,020 high-resolution images with detailed mask annotations, for model training and evaluation. Experimental results demonstrate that VLM-hyster substantially outperforms state-of-the-art AI models. Furthermore, extensive assessments by gynecologists, as well as multicentre and prospective validations, demonstrate VLM-hyster's robustness and generalizability. The results suggest that VLM-hyster earns considerable potential in enabling AI-assisted localization of surgical instruments and lesions in hysteroscopic surgeries. Code is available at https://github.com/viscom-tongji/VLM-hyster.
