Search papers, labs, and topics across Lattice.
This paper investigates the interaction between Knowledge Distillation (KD) and mixup during student training, revealing that applying mixup leads to a controlled mismatch that influences knowledge transfer. The authors find that while the teacher's supervisory signal is affected by distributional confusion, the student exhibits improved linearity in the vicinal region and surpasses traditional dark-knowledge transfer. Their approach consistently enhances student accuracy and reduces overconfidence across datasets like CIFAR and ImageNet, highlighting a significant calibration-accuracy trade-off influenced by temperature scaling.
Mixup-based distillation not only boosts student accuracy but also enhances uncertainty estimation, challenging the conventional view of knowledge transfer in deep learning.
Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs. Their interaction, however, remains poorly understood, particularly when mixup is applied only during student training. In this setting, the teacher is queried on inputs drawn from a vicinal distribution it never saw during training, a controlled mismatch whose effect on knowledge transfer has not been characterised. We show that this mismatch causes the teacher's supervisory signal to be dominated by distributional confusion rather than inter-class structure. Despite it, the student does not merely imitate the teacher: it independently acquires greater linearity in the vicinal region, a structural property that the teacher lacks, and goes beyond dark-knowledge transfer. KD with mixup consistently improves student accuracy and reduces overconfidence by an order of magnitude relative to the baseline, across CIFAR and ImageNet with varying-capacity teachers. Crucially, calibration propagates from teacher to student independently of accuracy transfer, and temperature scaling governs a measurable accuracy-calibration trade-off that becomes more pronounced under vicinal training. These results reframe mixup distillation not as a degraded version of standard KD, but as a richer transfer channel that simultaneously shapes discriminative performance, uncertainty estimation, and representational geometry.
