Search papers, labs, and topics across Lattice.
FireRedTTS3 introduces a unified framework for speech generation and editing that leverages semantically enriched continuous speech representations to mitigate error accumulation during autoregressive generation. By utilizing a frozen Audio Encoder as a semantic teacher, the model enhances text-speech alignment and stabilizes the generation process, achieving superior performance in both multilingual voice cloning and instruction-controlled voice design. Experimental results indicate that FireRedTTS3-Base and FireRedTTS3-Instruct outperform existing systems in terms of speech intelligibility, speaker similarity, and editing capabilities across multiple evaluation benchmarks.
Semantically enriched speech representations in FireRedTTS3 lead to unprecedented stability and fidelity in voice cloning and editing tasks.
Recent continuous autoregressive TTS models operate directly on continuous speech representations, preserving rich acoustic details while leveraging the instruction-following capabilities of text LLMs. This paradigm opens new possibilities for voice cloning, instruction-controlled voice design, and speech editing, but remains susceptible to error accumulation during autoregressive generation. Existing solutions often require additional semantic modules, multi-stage tokenizer training pipelines, or complex autoregressive architectures. In this work, we propose FireRedTTS3, a simple yet effective speech generation and editing framework that mitigates error accumulation at the representation level. Specifically, we leverage a frozen Audio Encoder trained on diverse speech understanding tasks as a semantic teacher to regularize the audio feature space. This improves text-speech alignment and stabilizes autoregressive generation while keeping the overall system simple. FireRedTTS3 provides two variants: FireRedTTS3-Base for multilingual and multi-dialect zero-shot voice cloning, and FireRedTTS3-Instruct for unified voice cloning, instruction-controlled voice design, and speech editing. Experiments show that FireRedTTS3-Base achieves the best average speech intelligibility and speaker similarity among compared systems on Seed-TTS-Eval and MiniMax-MLS-Test, while FireRedTTS3-Instruct outperforms competing systems on InstructTTSEval and Ming-Freeform-Audio-Edit. These results demonstrate that semantically enriched continuous speech representations, combined with a simple architecture, enable stable, controllable, and high-fidelity speech generation and editing. Code and models are available at https://github.com/FireRedTeam/FireRedTTS3.
