Search papers, labs, and topics across Lattice.
This paper introduces SwiftAudio, a novel one-step text-to-audio diffusion framework that distills knowledge from a pretrained diffusion teacher using only text captions, eliminating the need for paired audio data. By adapting Variational Score Distillation (VSD) and incorporating a temporal smoothness regularization objective, SwiftAudio achieves high-quality audio synthesis with significantly reduced inference latency. Experimental results on datasets like AudioCaps and Clotho show that SwiftAudio not only sets a new benchmark for one-step methods but also closely approaches the performance of traditional multi-step diffusion systems.
SwiftAudio achieves state-of-the-art text-to-audio synthesis with only text captions, challenging the need for paired audio data in model training.
Diffusion-based text-to-audio (TTA) models achieve impressive synthesis quality but suffer from high inference latency due to iterative multi-step denoising. Existing one-step approaches alleviate this issue but still rely on paired text--audio data during distillation. To address these limitations, we propose SwiftAudio, a one-step TTA framework that performs audio-free distillation from a pretrained diffusion teacher using only text captions. Specifically, we adapt Variational Score Distillation (VSD) to the audio domain and introduce a temporal smoothness regularization objective to encourage coherent latent audio representations. This design enables the student model to inherit the teacher's generative prior without requiring paired audio supervision and allows effective training with only approximately 45K captions. Experiments on AudioCaps and Clotho demonstrate that SwiftAudio achieves state-of-the-art performance among strict one-step methods and substantially narrows the gap to multi-step diffusion systems. Project page: https://swiftaudio.org/
