Search papers, labs, and topics across Lattice.
This paper introduces IF-Beta, a novel data pruning framework designed to enhance knowledge distillation (KD) by efficiently selecting training data through influence functions and a learnable sampling policy. The method addresses the computational overhead typically associated with KD by optimizing a bilevel objective that aligns with the distillation process, allowing for effective pruning without the need for extensive retraining. Experimental results on CIFAR-10/100 and ImageNet demonstrate that IF-Beta not only reduces data and compute requirements but also enables student models to outperform those trained on full datasets, highlighting its efficacy in resource-constrained environments.
IF-Beta allows student models to achieve superior performance with significantly less data and compute, challenging the traditional reliance on full datasets in knowledge distillation.
Knowledge Distillation (KD) is widely used to obtain compact models for efficient inference in resource-constrained environments. Yet the computational overhead of the distillation process itself is often overlooked, raising the question of whether a better student model can be obtained with less data and less compute via data pruning. However, existing data pruning methods are not designed for KD: some introduce substantial overhead, such as obtaining training dynamics through retraining, while others rely on heuristic selection rules that fail to capture what KD actually requires, often resulting in suboptimal subsets. To address these issues, we propose IF-Beta, an efficient data pruning framework that combines influence functions with a learnable sampling policy. Empirically, we first demonstrate that influence functions can serve as an effective and efficient estimator of sample impact in KD settings, where only a pretrained teacher is available. Building on this, our sampling policy is specifically parameterized by a Beta distribution, whose highly flexible two-parameter family allows the policy to adapt to diverse pruning regimes rather than being tied to fixed heuristic forms. Next, we formulate KD pruning as optimizing this policy through a bilevel objective, where the inner loop operates in the teacher feature space with a KD-aligned objective, enabling fast proxy training, while the outer loop updates the policy parameters to maximize distillation performance. This design ensures that IF-Beta is both computationally efficient and inherently aligned with the goals of KD. Extensive experiments on CIFAR-10/100 and ImageNet show that IF-Beta consistently outperforms other baselines across a wide range of pruning ratios. Remarkably, IF-Beta enables students trained on less data and less compute to surpass the performance of students distilled on the full dataset.
