Search papers, labs, and topics across Lattice.
This paper introduces a novel bootstrapped method for data selection in vision-language settings, leveraging a CLIP model that iteratively trains on a self-selected dataset. By balancing filtered, high-probability clean samples with diverse samples from the entire distribution, the method enhances the dataset's quality without requiring additional data or pre-trained models. The results demonstrate significant improvements in downstream performance, highlighting the effectiveness of the Self-Filtering approach in mitigating data noise at scale.
Iteratively training on a self-selected dataset can dramatically enhance vision-language model performance without the need for extra data or pre-training.
The availability of large amounts of clean data is paramount to training neural networks. However, at large scales, manual oversight is impractical, resulting in sizeable datasets that can be very noisy. Attempts to mitigate this obstacle to producing performant vision-language models have so far involved heuristics, curated reference datasets, and using pre-trained models. Here we propose a novel, bootstrapped method in which a CLIP model is trained on an evolving, self-selected dataset. This evolving dataset constitutes a balance of filtered, highly probable clean samples as well as diverse samples from the entire distribution. Our proposed Self-Filtering method iterates between training the model and selecting a subsequently improved data mixture. Training on vision-language datasets filtered by the proposed approach improves downstream performance without the need for additional data or pre-trained models.
Mila