Search papers, labs, and topics across Lattice.
Confucius4-TTS introduces a novel multilingual zero-shot text-to-speech system that eliminates the need for audio prompt transcripts, enabling both intra-lingual and cross-lingual voice cloning across 14 languages. By employing a two-stage architecture with a learnable speaker encoder and a conditional flow-matching module, the system achieves high intelligibility and speaker similarity, outperforming existing models on public benchmarks. Notably, it achieves an average word error rate (WER) of 3.73% on the CV3-Eval cross-lingual benchmark and ranks highest in human evaluations against recent systems.
Transcript-free voice cloning across 14 languages achieves unprecedented fidelity and intelligibility, setting a new standard for zero-shot TTS systems.
Recent advances in zero-shot text-to-speech (TTS) have substantially improved speech quality and voice cloning fidelity. However, many zero-shot TTS systems still depend on audio prompt transcripts at inference time. This dependency limits cross-lingual voice cloning, since in-the-wild reference audio is often untranscribed. In this technical report, we present Confucius4-TTS, a multilingual zero-shot TTS system that supports 14 languages and performs both intra-lingual and cross-lingual reference cloning without requiring transcripts of audio prompts. Confucius4-TTS follows a two-stage architecture, consisting of text-to-semantic (T2S) and semantic-to-acoustic (S2A) modules. The LLM-based T2S module uses a learnable speaker encoder to extract timbre features from self-supervised speech representations, and the conditional flow-matching S2A module converts the predicted semantic tokens into mel-spectrograms. The same model also supports continuation cloning when a reference transcript is available. Confucius4-TTS is trained on large-scale multilingual speech data. It achieves high intelligibility and speaker similarity on public benchmarks. On the CV3-Eval cross-lingual benchmark, Confucius4-TTS obtains an average WER of 3.73% across six directions. On our internal cross-lingual set, it achieves the best average overall rank in human evaluation among recent open-source and commercial systems. We release code, model checkpoints, and demos at https://github.com/netease-youdao/Confucius4-TTS.
