Search papers, labs, and topics across Lattice.
This paper introduces CADRE, a framework for the continual adaptation of medical vision-language models that addresses catastrophic forgetting and prior drift during updates for new imaging modalities. By integrating low-rank adaptation with a self-scaling, similarity-aware elastic weight consolidation term, CADRE effectively minimizes competence loss while maintaining alignment with the pretrained model's prior. In a controlled study involving breast cancer detection across three distinct imaging modalities, CADRE demonstrated a sevenfold reduction in forgetting compared to the best baseline, achieving superior accuracy and backward transfer metrics.
Forgetting in medical VLMs can be reduced by sevenfold, ensuring safer and more reliable adaptations to new imaging modalities.
Medical vision-language models (VLMs) such as BiomedCLIP generalize broadly, but adapting them to a clinical service is as much a safety problem as an accuracy one. Updating a deployed model for a new imaging modality can fail silently in two ways that harm patients: it can forget modalities it already handled (catastrophic forgetting), and it can drift from its trustworthy pretrained prior toward modality-specific shortcuts. We study parameter-efficient continual adaptation through these two properties rather than leaderboard accuracy, presenting CADRE: a frozen-backbone framework combining low-rank adaptation (LoRA) with an online, self-scaling, similarity-aware elastic weight consolidation term that bounds retained-competence loss, and an anchor-to-prior penalty bounding embedding drift from the frozen prior. Two short guarantees, a bound on total consolidation mass and a scale-invariance property, remove the scale-related sources of vanilla EWC's order fragility. Using breast cancer across three maximally dissimilar modalities (histopathology, ultrasound, chest radiography) as a controlled cross-modality stress test, under a multi-seed, multi-order protocol with paired significance testing and training approximately 0.23% of parameters, CADRE attains the highest accuracy, SPQ, and backward transfer and the lowest forgetting among adapting methods, reducing forgetting roughly sevenfold versus the strongest regularized baseline (0.075 to 0.011; paired p=0.023) and achieving positive backward transfer where every baseline is negative. We frame these as stability properties aligned with clinical-safety desiderata, not a deployment guarantee; robustness to distribution shift and adversarial inputs is out of scope.
