Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces DiffPT, a diffusion model-based pretraining strategy designed to improve classifier performance in low-data regimes by leveraging generative modeling principles. Unlike contrastive methods that excel on standard image benchmarks, DiffPT demonstrates superior efficacy on time-series and medical imaging datasets, outperforming various self-supervised baselines. The findings highlight diffusion models as a potent and versatile tool for data-efficient pretraining across diverse data modalities.
Diffusion models unlock superior classifier pretraining on medical imaging and time-series data, outperforming contrastive methods in low-data settings.
Machine learning with limited data is susceptible to overfitting, especially with modern deep neural networks. Prior to the introduction of AlexNet, reconstruction-based pretraining with auto-encoder and denoising auto-encoder were found to be beneficial, partially due to the limited size of the training sets at the time. Motivated by these traditional approaches, we propose DiffPT, a simple diffusion modeling approach to pretrain a given classifier backbone for limited domain-specific training data. We experiment on time-series data, standard natural-image benchmarks, and medical imaging datasets, and contrast the diffusion approach to a multitude of modern and traditional un/self-supervised methods. While contrastive pretraining performs better on CIFAR-10 and Tiny ImageNet in low-data regimes, DiffPT is particularly effective on medical imaging and time-series tasks, where it outperforms self-supervised baselines across a range of low-data regimes. Our results suggest that DiffPT is a simple and effective approach for data-efficient classifier pretraining on different modalities.