Search papers, labs, and topics across Lattice.
This paper introduces Early Intervention (EI), a framework that injects high-level semantic tokens from reference modalities into the target modality's embedding process early on, improving multimodal fusion in medical imaging. To address data scarcity and domain shift, they also propose Mixture of Low-varied-Ranks Adaptation (MoR), a parameter-efficient fine-tuning method for Vision Foundation Models (VFMs). Experiments on three medical imaging datasets demonstrate EI's superior performance compared to existing methods.
Injecting semantic information from related modalities early in the embedding process significantly boosts performance on multimodal medical image classification tasks.
Current methods for multimodal medical imaging based disease recognition face two major challenges. First, the prevailing "fusion after unimodal image embedding" paradigm cannot fully leverage the complementary and correlated information in the multimodal data. Second, the scarcity of labeled multimodal medical images, coupled with their significant domain shift from natural images, hinders the use of cutting-edge Vision Foundation Models (VFMs) for medical image embedding. To jointly address the challenges, we propose a novel Early Intervention (EI) framework. Treating one modality as target and the rest as reference, EI harnesses high-level semantic tokens from the reference as intervention tokens to steer the target modality's embedding process at an early stage. Furthermore, we introduce Mixture of Low-varied-Ranks Adaptation (MoR), a parameter-efficient fine-tuning method that employs a set of low-rank adapters with varied ranks and a weight-relaxed router for VFM adaptation. Extensive experiments on three public datasets for retinal disease, skin lesion, and keen anomaly classification verify the effectiveness of the proposed method against a number of competitive baselines.
