Search papers, labs, and topics across Lattice.
This paper introduces BioMedVR, a novel framework that enhances visual reprogramming for biomedical imaging by incorporating a Confusion Minimization Mechanism to address the challenges posed by class confusion in fine-grained medical scenarios. By leveraging confusion-aware attributes generated by large language models and employing a Mixture-of-Prompt Experts approach, BioMedVR enables few-shot adaptation of pretrained vision-language models, achieving significant improvements in accuracy and generalization across 18 diverse datasets. The results indicate that this method effectively reduces false-positive predictions, making it a critical advancement in the application of VLMs to biomedical contexts.
Misclassifications in biomedical imaging can be drastically reduced with BioMedVR's confusion-aware approach, leading to superior model performance in critical healthcare applications.
Recent advances in vision-language models (VLMs) such as CLIP have demonstrated strong generalization across natural-image domains. However, adapting these models to biomedical imaging is non-trivial: full-model fine-tuning is computationally expensive, while medical data are often scarce and exhibit subtle, fine-grained inter-class differences, making parameter-efficient adaptation particularly critical. Visual Reprogramming (VR) offers a parameter-efficient alternative by injecting learnable perturbations into the input space, but existing VR approaches for VLMs mainly focus on positive class prompts and overlook confusing negatives, leading to miscalibrated predictions in fine-grained medical scenarios. We present BioMedVR, the first VR-based framework for biomedical imaging, enabling few-shot adaptation of pretrained VLMs through compact learnable VR modules. To mitigate class confusion, we introduce a Confusion Minimization Mechanism that leverages LLM-generated confusion-aware attributes together with a Confusion-Suppression Loss to explicitly reduce false-positive alignment. Moreover, the designed Mixture-of-Prompt Experts combines a positive expert for main-class discrimination and a negative expert for confusion suppression, balanced via adaptive gating. Extensive experiments on 18 datasets, including 11 biomedical datasets and 7 natural image benchmarks, demonstrate that BioMedVR achieves superior accuracy and generalization, effectively bridging VR and VLMs in biomedical domains.
