Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the impact of textual information on pixel-level predictions in medical image segmentation using pretrained vision-language models (VLMs). By analyzing various fusion strategies and introducing the Evidence Decoupling Decoder (EDD), the authors reveal that segmentation performance is largely insensitive to the choice of fusion module, yet the sensitivity to text perturbation varies significantly across datasets. Notably, while some datasets show a strong reliance on text, others indicate that text primarily influences predictions through global semantic modulation rather than spatial localization.
Textual input can drastically alter segmentation outcomes, with some datasets showing catastrophic performance drops when text is removed, while others remain largely unaffected.
Pretrained vision-language models (VLMs) have shown promising performance in medical image segmentation by incorporating clinical text. However, it remains unclear how much textual information actually contributes to pixel-level predictions. In this work, we systematically investigate the role of text in multimodal medical image segmentation. We first analyze several commonly used fusion strategies and find that segmentation performance is largely insensitive to the choice of fusion module. To further understand modality interactions, we propose an Evidence Decoupling Decoder (EDD) based on evidential deep learning and deep supervision. EDD serves as an internal representation analysis tool that decomposes image evidence and text-modulated evidence throughout the decoding process while maintaining competitive segmentation performance. Experimental results show that the sensitivity to text perturbation varies substantially across datasets. On BUSI and BTMRI, removing text causes catastrophic performance drops, indicating strong model reliance on textual input. On ISIC and Kvasir-SEG, text exerts relatively marginal influence. We further find that text affects predictions mainly through global semantic modulation rather than independent spatial localization, and that the specific semantic components driving text sensitivity differ across datasets. These findings provide a deeper understanding of modality interaction in multimodal medical image segmentation and offer practical insights for future model design.