Search papers, labs, and topics across Lattice.
This paper introduces a distillation-based pretraining framework for multiple instance learning (MIL) networks, utilizing two slide-level foundation models, TITAN and CARE, to enhance the learning process. By implementing an angular dispersion normalized distillation loss, the framework effectively balances the supervision from multiple teachers, leading to improved initialization for downstream tasks. Experimental results across 15 benchmark datasets demonstrate that this approach significantly enhances performance in linear probing and few-shot scenarios compared to training from scratch, while preserving the efficiency of lightweight MIL models.
Pretraining MIL networks with knowledge distillation from foundation models boosts performance and stability, especially in few-shot scenarios.
Multiple instance learning (MIL) has become the main paradigm for whole-slide image (WSI) analysis in computational pathology. However, existing MIL aggregators are still typically trained from scratch for each downstream task, relying on limited slide-level labels to learn both aggregation mechanisms and downstream discriminative representations simultaneously. As a result, they often suffer from unstable optimization, overfitting, and limited transferability. Similar to pretrained ResNet and Vision Transformer models in natural image learning, MIL also requires reusable pretrained initialization. However, high-quality slide-level pretraining data remain scarce, and MIL models are usually lightweight and weakly supervised, making large-scale pretraining difficult in practice. To address this challenge, we propose a distillation-based pretraining framework for MIL, which leverages two slide-level foundation models, TITAN and CARE, as teachers to transfer their representational knowledge into a diverse set of MIL architectures. To effectively balance supervision from different teachers, we further introduce an angular dispersion normalized distillation loss. The distilled weights are then used as initialization for downstream adaptation. We conduct systematic evaluations on 15 benchmark datasets under both linear probing and full-parameter fine-tuning, and further validate its advantages in few-shot scenarios. Experimental results show that pretraining generally improves MIL aggregators over from scratch training, especially in linear-probing and few-shot settings, while maintaining the computational efficiency of lightweight MIL models. Code is available at https://github.com/fu0201/MIL_Pretrained.
Tsinghua AI