Search papers, labs, and topics across Lattice.
This paper introduces LUX, a novel graph-conditioned vision-language architecture designed for explainable endoscopic captioning in ulcerative colitis, addressing the limitations of traditional models that rely on global visual embeddings. By constructing lesion-centric scene graphs from Grad-CAM and CBAM activation maps, LUX enhances the interpretability and reliability of automated analyses by allowing generated captions to focus on specific pathological regions. The model demonstrates significant performance improvements over existing medical captioning benchmarks, particularly in reducing hallucinated findings and enhancing lesion-level grounding.
LUX achieves unprecedented alignment between generated captions and localized pathological evidence, drastically reducing clinical hallucinations in endoscopic analysis.
The interpretation of endoscopic imagery in ulcerative colitis is complex and subjective, with variability in human assessment and subtle mucosal inflammation. Although deep learning has advanced automated analysis, most vision-language models rely on global visual embeddings that overlook the localized and relational nature of pathological evidence, limiting clinical reliability and interpretability. We introduce LUX (Lesion-aware Unified eXplainable captioning), a graph-conditioned vision-language architecture for explainable endoscopic image captioning. LUX constructs a lesion-centric scene graph from Grad-CAM and CBAM activation maps, representing pathological regions as nodes and encoding their spatial and clinical relationships. These graph embeddings are integrated into the cross-attention layers of a T5 decoder, enabling generated words to attend to specific lesion nodes rather than only to global image features. This provides direct alignment between linguistic content and pathological evidence, supporting token-level interpretability and relational reasoning. LUX outperforms strong baseline and state-of-the-art medical captioning models across BLEU, METEOR, ROUGE-L, and CIDEr, with particularly strong gains in CIDEr. It also reduces hallucinated clinical findings and improves lesion-level grounding through stronger correspondence between generated tokens and localized pathological regions.
