Search papers, labs, and topics across Lattice.
This study introduces EndoCLIP, a vision-language foundation model specifically designed for colonoscopy, which was trained on 125,756 lesion-level image-text pairs extracted from over 280,000 routine reports. By effectively linking clinical findings to corresponding images, EndoCLIP significantly improves performance in tasks such as image-text retrieval and clinical classification, outperforming existing general-purpose and biomedical models. Notably, in a blinded study, EndoCLIP's performance in benign-versus-malignant classification approaches that of expert endoscopists, highlighting its potential to enhance clinical decision-making through better integration of visual and textual data.
EndoCLIP can match expert endoscopists in classifying lesions, showcasing the transformative potential of linking clinical findings to images in colonoscopy.
Vision-language models remain underused in colonoscopy despite the rich expert descriptions recorded in routine reports. These reports document lesion appearance, size and location but summarise entire procedures rather than caption individual frames, leaving clinical findings only weakly linked to the corresponding images. Here we develop EndoCLIP, a colonoscopy vision-language foundation model trained on 125,756 lesion-level image-text pairs progressively recovered from 280,476 routine colonoscopy records. Across lesion-level image-text retrieval, structured report generation and six multi-centre clinical classification tasks, EndoCLIP outperforms general-purpose and biomedical vision-language encoders in both zero-shot and linear-probe settings. On benign-versus-malignant classification, its linear probe approaches the performance of expert readers in a blinded study involving 12 endoscopists. These results suggest that recovering finding-to-frame correspondence can transform routine documentation into scalable supervision, enabling clinical targets to be specified in language rather than separately annotated for each task.
Microsoft Research