Search papers, labs, and topics across Lattice.
This paper introduces KG-CMI, a Knowledge Graph enhanced Cross-Mamba Interaction framework, to improve medical visual question answering (Med-VQA). KG-CMI leverages a knowledge graph to associate lesion features in medical images with diagnostic criteria and uses a free-form answer enhanced multi-task learning module to improve open-ended question answering. Experiments on VQA-RAD, SLAKE, and OVQA datasets demonstrate that KG-CMI outperforms existing state-of-the-art methods.
Medical VQA gets a boost: Mamba-based models enhanced with knowledge graphs now outperform existing methods by better associating lesion features with diagnostic criteria.
Medical visual question answering (Med-VQA) is a crucial multimodal task in clinical decision support and telemedicine. Recent methods fail to fully leverage domain-specific medical knowledge, making it difficult to accurately associate lesion features in medical images with key diagnostic criteria. Additionally, classification-based approaches typically rely on predefined answer sets. Treating Med-VQA as a simple classification problem limits its ability to adapt to the diversity of free-form answers and may overlook detailed semantic information in those answers. To address these challenges, we propose a knowledge graph enhanced cross-Mamba interaction (KG-CMI) framework, which consists of a fine-grained cross-modal feature alignment (FCFA) module, a knowledge graph embedding (KGE) module, a cross-modal interaction representation (CMIR) module, and a free-form answer enhanced multi-task learning (FAMT) module. The KG-CMI learns cross-modal feature representations for images and texts by effectively integrating professional medical knowledge through a graph, establishing associations between lesion features and disease knowledge. Moreover, FAMT leverages auxiliary knowledge from open-ended questions, improving the model's capability for open-ended Med-VQA. Experimental results demonstrate that KG-CMI outperforms existing state-of-the-art methods on three Med-VQA datasets, i.e., VQA-RAD, SLAKE, and OVQA. Additionally, we conduct interpretability experiments to further validate the framework's effectiveness.
