Search papers, labs, and topics across Lattice.
This paper presents a novel framework for medical image captioning that enhances clinical alignment by separating training-time and inference-time processes. By employing a multi-encoder design with BioMedCLIP and SigLIP2, along with a LLaMA-based decoder, the authors integrate auxiliary learning for UMLS concept prediction and implement a reranking strategy at inference to improve caption quality. The results demonstrate that this approach significantly enhances the generation of clinically relevant captions, promoting reliability in medical diagnostics even with limited data.
Jointly leveraging selection-based and reinforcement-learning-based alignment can transform medical image captioning into a more trustworthy diagnostic tool.
Medical image captioning is a technique that accelerates early-stage diagnostic workflows and enhances the interpretability of medical diagnostic AI systems. However, unlike general image captioning, clinically reliable captioning remains challenging due to grayscale-based modalities, subtle anatomical cues, specialized medical phrasing, and variations in data quality. Despite recent advances in large vision-language models, fluent outputs do not necessarily guarantee sufficient alignment with clinical concept spaces or evaluation criteria. To address this issue, we propose a framework that strengthens clinical alignment by separating and enhancing training-time alignment and inference-time alignment. We build a medical image captioning pipeline that integrates single/dual vision encoders based on BioMedCLIP and SigLIP2, a Q-Former, and a LLaMA-based decoder, and examine the contribution of auxiliary learning for UMLS concept/type prediction. At inference, we apply single-embedding-based reranking to select the best caption among candidates, while at training we introduce MedPAIR-SCST, which combines clinically relevant rewards to shift the generative distribution toward improved clinical alignment. Our experiments show that complementary visual representations with a multi-encoder design and concept-level auxiliary learning help preserve clinically meaningful information. Furthermore, inference-time reranking provides a practical way to improve semantic and clinical alignment without additional training, whereas MedPAIR-SCST goes beyond selection by directly improving the model's distribution to generate more consistent and clinically grounded captions. These findings suggest that jointly leveraging selection-based alignment and reinforcement-learning-based alignment can promote more trustworthy medical image captioning even in data-constrained settings.
