Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores the repurposing of a conventional speech classifier as a backbone for guided diffusion-based speech generation, addressing the inefficiencies of requiring separate models for classification and diffusion. By utilizing a frozen noise-conditioned classifier and attaching a lightweight subnetwork that leverages intermediate representations, the authors train this subnetwork under a Denoising Score Matching objective. The key finding reveals that this approach not only maintains high speech quality but also significantly reduces memory usage and computational costs compared to traditional methods.
Repurposing a speech classifier for diffusion generation slashes memory usage while preserving high-quality output, challenging the need for separate models.
Classifier guidance is a way to control diffusion generation by using a noise-conditioned classifier to steer the sampling process toward a target class. One drawback of classifier guidance is that it requires two separately trained models: a classifier and a diffusion model. We therefore study a more compact alternative in which a conventionally trained speech classifier is repurposed as the backbone for diffusion generation. Starting from a frozen noise-conditioned classifier in log-Mel space, we attach a lightweight subnetwork that reuses intermediate classifier representations and train only this subnetwork under a Denoising Score Matching objective. Our work shows that a pretrained classifier can be repurposed for conditional generation, providing an appealing bridge between discriminative modeling and conditional speech synthesis resulting in high speech quality within a single-backbone model, with reduced memory footprint and computational cost.