Search papers, labs, and topics across Lattice.
The paper introduces MedCLIPSeg, a framework that adapts CLIP for medical image segmentation by using patch-level CLIP embeddings and probabilistic cross-modal attention to enable bidirectional interaction between image and text tokens. This approach incorporates a soft patch-level contrastive loss to improve semantic learning across textual prompts, enhancing data efficiency and domain generalizability. Experiments across 16 datasets show that MedCLIPSeg achieves superior accuracy, efficiency, and robustness compared to existing methods, while also providing interpretable uncertainty maps.
MedCLIPSeg unlocks data-efficient and generalizable medical image segmentation by probabilistically fusing CLIP's vision-language understanding at the patch level, outperforming existing methods across a wide range of datasets and modalities.
Medical image segmentation remains challenging due to limited annotations for training, ambiguous anatomical features, and domain shifts. While vision-language models such as CLIP offer strong cross-modal representations, their potential for dense, text-guided medical image segmentation remains underexplored. We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation. Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens and explicit modeling of predictive uncertainty. Together with a soft patch-level contrastive loss that encourages more nuanced semantic learning across diverse textual prompts, MedCLIPSeg effectively improves data efficiency and domain generalizability. Extensive experiments across 16 datasets spanning five imaging modalities and six organs demonstrate that MedCLIPSeg outperforms prior methods in accuracy, efficiency, and robustness, while providing interpretable uncertainty maps that highlight local reliability of segmentation results. This work demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation.
