Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a noise-aware framework for medical visual question answering (Med-VQA) that integrates a denoising autoencoder to enhance the robustness of visual representations before they are processed by large language models (LLMs). By pretraining the autoencoder to reconstruct clean embeddings from noisy inputs, the approach effectively mitigates the impact of irrelevant visual changes, leading to improved performance on the SLAKE and PathVQA benchmarks. The results demonstrate that the proposed method not only maintains competitive performance on clean inputs but also significantly enhances robustness against noise, highlighting the importance of addressing visual representation quality in Med-VQA systems.
Robust visual representations can boost Med-VQA performance by up to 20% in noisy environments, challenging the status quo of current methods.
Medical visual question answering (Med-VQA) has strong potential for clinical decision support by enabling AI models to interpret medical images and answer clinically relevant queries. Recent approaches typically connect off-the-shelf vision encoders with large language models (LLMs) through lightweight mapping networks to reduce computational cost. However, these methods often overlook the importance of handling noise and small irrelevant changes in visual representations. To address these challenges, we propose a noise-aware Med-VQA framework that incorporates a denoising autoencoder before visual embeddings are mapped into the input space of an LLM. The denoising autoencoder is pretrained to reconstruct clean visual embeddings from corrupted inputs, encouraging the model to learn robust visual representations that are less sensitive to noise. The resulting embeddings are then projected into the language model embedding space using a multi-layer perceptron (MLP), forming visual prefix tokens that provide image information to the LLM. To enable efficient adaptation without full retraining, we employ parameter-efficient fine-tuning using low-rank adaptation (LoRA). The proposed method is evaluated on the SLAKE and PathVQA benchmarks. Experimental results show improved robustness to noisy input embeddings while maintaining competitive clean performance across multiple evaluation criteria. These findings suggest that learning more robust visual representations can enhance Med-VQA performance and robustness.