Search papers, labs, and topics across Lattice.
This paper introduces a novel framework that integrates self-supervised learning with diffusion models for speech enhancement by conditioning the model on wav2vec 2.0 features. By employing Feature-wise Linear Modulation (FiLM) at the U-Net bottleneck, the approach leverages phonetic representations from degraded speech to guide the reverse diffusion process. The method achieves a notable improvement of 0.4 in PESQ scores over unconditioned baselines, demonstrating the efficacy of self-supervised representations in enhancing speech quality.
Self-supervised representations can boost diffusion-based speech enhancement, yielding a significant 0.4 improvement in PESQ scores.
Diffusion models show potential for speech enhancement but lack linguistic guidance. We condition a diffusion-based model on wav2vec 2.0 features from noisy input, injected at the U-Net bottleneck via Feature-wise Linear Modulation (FiLM). Phonetic representations from wav2vec 2.0 features of degraded speech, anchor the reverse diffusion process. While a frozen wav2vec 2.0 encoder extracts features, a learned FiLM generator produces scale and shift parameters modulating the bottleneck with minimal overhead. Motivated by the optimal Bayesian causal estimator under a linear-Gaussian state-space model, FiLM coefficients are aggregated via exponential smoothing for temporal compression. Evaluation on VoiceBank-DEMAND and LibriMix shows competitive performance against the unconditioned baseline in PESQ, STOI, SI-SDR and DNSMOS. We consistently record an improvement of 0.4 on PESQ score, suggesting self-supervised representations effectively condition diffusion-based speech enhancement.
