Search papers, labs, and topics across Lattice.
This paper introduces CookVoice, a unified framework for multi-modal human voice generation that effectively integrates speech and singing voice tasks while allowing for fine-grained control over content, prosody, and style. By employing a flexible alignment strategy that maps control signals to spectrogram frames, CookVoice achieves high-quality voice generation with enhanced controllability across various applications such as text-to-speech and voice mimicry. Experimental results indicate that CookVoice matches the performance of larger models while maintaining a compact architecture with only 43.51 million parameters and efficient inference capabilities.
CookVoice achieves high-quality, style-controllable voice generation with a fraction of the parameters, outperforming larger models in efficiency and flexibility.
Human voice generation has made rapid progress in speech generation, singing voice generation, voice cloning, and voice editing. However, most existing systems are designed for specific tasks and often rely on task-dependent architectures, control signals, or autoregressive decoding, limiting fine-grained controllability and inference efficiency. In this paper, we propose CookVoice, a unified framework for multimodal, multi-style, and multi-task human voice generation. CookVoice decomposes the human voice into three key factors: content, prosody, and style, enabling both speech and singing voice generation within a unified model. To achieve precise and flexible controllability, we design a flexible alignment strategy that maps text, style, and prosody control signals onto the frame-level of spectrogram. This design allows CookVoice to support a wide range of tasks, including text-to-speech, text-to-singing voice, style-controllable generation, voice mimicry, voice conversion, and voice editing. Experimental results show that CookVoice achieves generation quality comparable to existing Text-to-Speech and text-to-singing voice baselines, while providing stronger style and prosody controllability. Moreover, CookVoice achieves comparable performance to large-scale baselines with only 43.51 million parameters and efficient inference using as few as 4 ODE steps, making it a practical solution for real-world human voice generation applications. Demo page is available at https://haoweilou.github.io/CookVoice/.
