Search papers, labs, and topics across Lattice.
This paper addresses the performance degradation of dataset distillation (DD) methods on long-tailed datasets by identifying issues with heuristic distribution discrepancy measures and uniform treatment of imbalanced classes. They propose Class-Aware Spectral Distribution Matching (CSDM), which aligns distributions via the spectrum of a kernel function, mapping samples to frequency space and using Spectral Distribution Distance (SDD). CSDM also employs amplitude-phase decomposition to prioritize realism in tail classes, achieving significant performance improvements and stability on long-tailed CIFAR-10.
Dataset distillation gets a boost on long-tailed data with CSDM, which uses spectral distribution matching to prioritize realism in tail classes and achieves a 14% improvement over SOTA methods.
Dataset distillation (DD) aims to compress large-scale datasets into compact synthetic counterparts for efficient model training. However, existing DD methods exhibit substantial performance degradation on long-tailed datasets. We identify two fundamental challenges: heuristic design choices for distribution discrepancy measure and uniform treatment of imbalanced classes. To address these limitations, we propose Class-Aware Spectral Distribution Matching (CSDM), which reformulates distribution alignment via the spectrum of a well-behaved kernel function. This technique maps the original samples into frequency space, resulting in the Spectral Distribution Distance (SDD). To mitigate class imbalance, we exploit the unified form of SDD to perform amplitude-phase decomposition, which adaptively prioritizes the realism in tail classes. On CIFAR-10-LT, with 10 images per class, CSDM achieves a 14.0% improvement over state-of-the-art DD methods, with only a 5.7% performance drop when the number of images in tail classes decreases from 500 to 25, demonstrating strong stability on long-tailed data.
Tsinghua AI