Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces WhispSynth, a large-scale multilingual whispered speech corpus generated using a novel pitch-free DDSP-based TTS pipeline. This pipeline leverages a new WhispNJU dataset and other resources to generate 118 hours of high-fidelity whispered speech from 479 speakers, preserving vocal timbre and linguistic content. Fine-tuning with WhispSynth allows their CosyWhisper model to achieve speech naturalness comparable to real whispered speech, demonstrating the corpus's quality and utility.
A new synthetic whispered speech corpus, WhispSynth, closes the data gap in text-to-whisper research by achieving naturalness scores on par with real recordings.
Whisper generation is constrained by the difficulty of data collection. Because whispered speech has low acoustic amplitude, high-fidelity recording is challenging. In this paper, we introduce WhispSynth, a large-scale multilingual corpus constructed via a novel high-fidelity generative framework. Specifically, we propose a pipeline integrating Differentiable Digital Signal Processing (DDSP)-based pitch-free method with Text-to-Speech (TTS) models. This framework refines a comprehensive collection of resources, including our newly constructed WhispNJU dataset, into 118 hours of high-fidelity whispered speech from 479 speakers. Unlike standard synthetic or noisy real data, our data engine faithfully preserves source vocal timbre and linguistic content while ensuring acoustic consistency, providing a robust foundation for text-to-whisper research. Experimental results demonstrate that WhispSynth exhibits significantly higher quality than existing corpora. Moreover, our CosyWhisper, tuned with WhispSynth, achieves speech naturalness on par with ground-truth samples. The official implementation and related resources are available at https://github.com/tan90xx/cosywhisper.