Search papers, labs, and topics across Lattice.
This paper addresses the challenge of excessive storage requirements for soft labels in large-scale dataset distillation, which can be 30-40x larger than the condensed images on ImageNet-1K. The authors introduce Label Pruning and Quantization for Large-scale Distillation (LPQLD) to enhance both image and supervision diversity, achieving a remarkable reduction in soft label storage by 78x on ImageNet-1K and 500x on ImageNet-21K while simultaneously improving accuracy by up to 7.2% and 2.8%, respectively. Extensive experiments confirm the effectiveness of LPQLD across various network architectures and distillation methods, showcasing its potential for more efficient dataset compression.
Reducing soft label storage by up to 500x while boosting accuracy could revolutionize large-scale dataset distillation.
Large-scale dataset distillation requires storing auxiliary soft labels that can be 30-40x larger on ImageNet-1K and 200x larger on ImageNet-21K than the condensed images, undermining the goal of dataset compression. We identify two fundamental issues necessitating such extensive labels: (1) insufficient image diversity, where high within-class similarity in synthetic images requires extensive augmentation, and (2) insufficient supervision diversity, where limited variety in supervisory signals during training leads to performance degradation at high compression rates. To address these challenges, we propose Label Pruning and Quantization for Large-scale Distillation (LPQLD). We enhance image diversity via class-wise batching and batch-normalization supervision during synthesis. For supervision diversity, we introduce Label Pruning with Dynamic Knowledge Reuse to improve label-per-augmentation diversity, and Label Quantization with Calibrated Student-Teacher Alignment to improve augmentation-per-image diversity. Our approach reduces soft label storage by 78x on ImageNet-1K and 500x on ImageNet-21K while improving accuracy by up to 7.2% and 2.8%, respectively. Extensive experiments validate the superiority of LPQLD across different network architectures and dataset distillation methods. Code is available at https://github.com/he-y/soft-label-pruning-quantization-for-dataset-distillation.
