Search papers, labs, and topics across Lattice.
This paper introduces SQuaT, a novel framework that integrates self-supervised knowledge distillation with quantization-aware training to address the challenges of deploying quantized models without training labels. By applying the quantization parameters of the student model to the teacher's features, SQuaT eliminates the inherent lower bound on distillation loss observed in previous methods, leading to significant performance improvements. Comprehensive experiments reveal that SQuaT consistently outperforms existing baselines, particularly in extreme low-bit quantization scenarios, demonstrating its versatility across various architectures and settings.
Eliminating the lower bound on distillation loss could redefine the effectiveness of quantized models in label-scarce environments.
Quantization-Aware Training (QAT) enables the deployment of quantized models with minimal accuracy degradation. However, in practical scenarios, training labels are often unavailable due to privacy, copyright, or cost constraints. Knowledge Distillation (KD) is a common approach to address this challenge, but we observe that prior work combining QAT with KD suffers from a fundamental limitation: during distillation, the range mismatch between the teacher and the quantized student model induces an unattainable residual, resulting in an irreducible lower bound on the distillation loss. Motivated by this observation, we propose SQuaT (Student-Aware Quantized Teacher Features), a label-free QAT framework with KD that theoretically eliminates this lower bound by applying the student's quantization parameters to quantize the teacher's features during distillation. Through comprehensive experiments across diverse settings, we demonstrate that SQuaT consistently outperforms strong baselines, with particularly pronounced gains in extreme low-bit (e.g., 1- and 2-bit) settings. Furthermore, extensive evaluations across various model design choices show that our approach does not rely on specific architectural assumptions, making it broadly applicable across diverse architectures and quantization settings. The source code is available at https://github.com/lcdbsa522/SQuaT.
