Search papers, labs, and topics across Lattice.
The paper introduces Exploration-Exploitation Distillation (E^2D), a novel approach to accelerate large-scale dataset distillation by minimizing redundant computations. E^2D uses full-image initialization followed by a two-phase optimization strategy: an exploration phase with uniform updates to identify high-loss regions, and an exploitation phase focusing updates on those regions. Experiments on ImageNet-1K and ImageNet-21K demonstrate that E^2D achieves state-of-the-art accuracy with significant speedups (18x on ImageNet-1K and 4.3x on ImageNet-21K) compared to existing methods.
Dataset distillation can be sped up by 18x on ImageNet-1K without sacrificing accuracy by focusing optimization on high-loss regions.
Dataset distillation compresses the original data into compact synthetic datasets, reducing training time and storage while retaining model performance, enabling deployment under limited resources. Although recent decoupling-based distillation methods enable dataset distillation at large-scale, they continue to face an efficiency gap: optimization-based decoupling methods achieve higher accuracy but demand intensive computation, whereas optimization-free decoupling methods are efficient but sacrifice accuracy. To overcome this trade-off, we propose Exploration-Exploitation Distillation (E^2D), a simple, practical method that minimizes redundant computation through an efficient pipeline that begins with full-image initialization to preserve semantic integrity and feature diversity. It then uses a two-phase optimization strategy: an exploration phase that performs uniform updates and identifies high-loss regions, and an exploitation phase that focuses updates on these regions to accelerate convergence. We evaluate E^2D on large-scale benchmarks, surpassing the state-of-the-art on ImageNet-1K while being 18x faster, and on ImageNet-21K, our method substantially improves accuracy while remaining 4.3x faster. These results demonstrate that targeted, redundancy-reducing updates, rather than brute-force optimization, bridge the gap between accuracy and efficiency in large-scale dataset distillation. Code is available at https://github.com/ncsu-dk-lab.
