Search papers, labs, and topics across Lattice.
This paper introduces Contrastive Audio-Aware Distillation (CAAD), a novel framework that integrates contrastive reasoning from teacher models into student models to enhance the performance of Speech Language Models while mitigating the drawbacks of high parameter counts and inference latency. By employing a synchronized teacher-forcing strategy, CAAD enables efficient full-sequence generation of contrastive distributions, significantly reducing the computational overhead typically associated with dual-path token-by-token distillation. The results demonstrate an approximately 8% relative improvement over standard knowledge distillation on the Dynamic-SUPERB benchmark and a reduction in linguistic bias on the MCR-BENCH dataset.
CAAD achieves an 8% performance boost in speech language models while slashing inference latency and linguistic bias.
Speech Language Models achieve reasoning capabilities, but are often hindered by massive parameter counts and a tendency to prioritize linguistic priors over acoustic features. While contrastive decoding enhances grounding by contrasting audio-aware and text-only logits, it increases inference latency. We propose Contrastive Audio-Aware Distillation (CAAD), a framework that internalizes the teacher's contrastive reasoning into the student model's weights. To overcome the high computational training overhead in the dual-path token-by-token contrastive distillation process, we introduce a synchronized teacher-forcing strategy. Anchored by unified Pseudo-Ground Truths, this mechanism enables simultaneous full-sequence generation of the teacher's contrastive distributions, allowing student to distill the audio-aware signal efficiently. Overall, CAAD yields a ~8% relative gain over standard knowledge distillation on Dynamic-SUPERB and successfully reduces linguistic bias in MCR-BENCH.
