Search papers, labs, and topics across Lattice.
The paper introduces WhispSynth, a large-scale multilingual whispered speech corpus generated using a novel pipeline that combines Differentiable Digital Signal Processing (DDSP) with Text-to-Speech (TTS) models. This pipeline addresses the challenge of collecting high-fidelity whispered speech data by preserving vocal timbre and linguistic content while ensuring acoustic consistency. CosyWhisper, a model fine-tuned on WhispSynth, achieves speech naturalness comparable to real whispered speech, demonstrating the corpus's high quality and utility for text-to-whisper research.
Forget noisy recordings: WhispSynth offers a new path to high-fidelity whispered speech synthesis using a clever pitch-free generative framework that rivals real data.
Whisper generation is constrained by the difficulty of data collection. Because whispered speech has low acoustic amplitude, high-fidelity recording is challenging. In this paper, we introduce WhispSynth, a large-scale multilingual corpus constructed via a novel high-fidelity generative framework. Specifically, we propose a pipeline integrating Differentiable Digital Signal Processing (DDSP)-based pitch-free method with Text-to-Speech (TTS) models. This framework refines a comprehensive collection of resources, including our newly constructed WhispNJU dataset, into 118 hours of high-fidelity whispered speech from 479 speakers. Unlike standard synthetic or noisy real data, our data engine faithfully preserves source vocal timbre and linguistic content while ensuring acoustic consistency, providing a robust foundation for text-to-whisper research. Experimental results demonstrate that WhispSynth exhibits significantly higher quality than existing corpora. Moreover, our CosyWhisper, tuned with WhispSynth, achieves speech naturalness on par with ground-truth samples. The official implementation and related resources are available at https://github.com/tan90xx/cosywhisper.
