Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study advances gastrointestinal (GI) endoscopic visual question answering (VQA) by employing a multi-task learning approach that integrates auxiliary grounding and description tasks with minimal additional annotation. By leveraging expert-annotated polyp masks and weak supervision from a GI-domain pretrained classifier, the authors fine-tune three small vision-language models (VLMs) to enhance both answer accuracy and the alignment of answer tokens with relevant image regions. The results demonstrate significant accuracy improvements and better interpretability of model responses, validating the effectiveness of the proposed method on both in-distribution and out-of-distribution datasets.
Achieving improved accuracy and interpretability in GI endoscopic VQA by aligning model answers with visual evidence through innovative multi-task learning.
Gastrointestinal (GI) endoscopic image analysis has shifted from single-label classification toward visual question answering (VQA), where a model must answer free-form clinical questions about an image. While recent vision-language models (VLMs) achieve promising answer accuracy on this task, clinical adoption also requires the model's internal representations to reflect the visual evidence behind its answers. We propose a simple multi-task fine-tuning recipe that constructs auxiliary grounding and description tasks from an existing VQA dataset with minimal additional annotation: expert-annotated polyp masks are reused directly, while a GI-domain pretrained classifier with Grad-CAM localization provides weak supervision for finding categories that lack ground-truth masks. Three small VLM backbones are fine-tuned with low-rank adaptation under matched VQA-only and multi-task recipes on Kvasir-VQA-x1, and we show consistent accuracy gains together with improved implicit alignment between answer tokens and the relevant image region, evaluated on both in-distribution and out-of-distribution data.