Search papers, labs, and topics across Lattice.
The paper introduces an entropy-adaptive online model merging strategy for handling heterogeneous domain shifts in medical imaging, where models are fine-tuned on private, domain-specific data. They demonstrate the limitations of mean averaging in such scenarios and propose a method that leverages target information through batch-specific merging coefficients derived from forward passes. Their approach decouples the encoder and classification head, merging them with separate coefficients to mitigate encoder-classifier mismatch, achieving consistent performance gains across diverse datasets while maintaining single-model inference.
Forget simple averaging: this entropy-adaptive model merging strategy unlocks robust performance in medical imaging by intelligently combining models trained on diverse, private clinical datasets.
Model merging under unseen test-time distribution shifts often renders naive strategies, such as mean averaging unreliable. This challenge is especially acute in medical imaging, where models are fine-tuned locally at clinics on private data, producing domain-specific models that differ by scanner, protocol, and population. When deployed at an unseen clinical site, test cases arrive in unlabeled, non-i.i.d. batches, and the model must adapt immediately without labels. In this work, we introduce an entropy-adaptive, fully online model-merging method that yields a batch-specific merged model via only forward passes, effectively leveraging target information. We further demonstrate why mean merging is prone to failure and misaligned under heterogeneous domain shifts. Next, we mitigate encoder classifier mismatch by decoupling the encoder and classification head, merging with separate merging coefficients. We extensively evaluate our method with state-of-the-art baselines using two backbones across nine medical and natural-domain generalization image classification datasets, showing consistent gains across standard evaluation and challenging scenarios. These performance gains are achieved while retaining single-model inference at test-time, thereby demonstrating the effectiveness of our method.
