Search papers, labs, and topics across Lattice.
This paper introduces a fully end-to-end (E2E) optimization framework for text-to-speech (TTS) systems that integrates the training of the speech tokenizer, large language model (LLM), flow-matching (FM) model, and a reward model (RM) into a unified process. By jointly optimizing these components through multi-task objectives, the authors enhance the discrete speech token space to better capture relevant acoustic and semantic information, leading to improved performance. Experimental results demonstrate that this E2E approach achieves a new state-of-the-art word error rate (WER) of 0.78% on the Seed-TTS-Eval benchmark, significantly outperforming traditional cascaded pipelines.
A unified end-to-end training framework for TTS systems achieves a groundbreaking 0.78% word error rate, setting a new standard in the field.
Recent state-of-the-art (SOTA) text-to-speech (TTS) systems typically adopt a cascaded pipeline consisting of a speech tokenizer, an autoregressive large language model (LLM), and a diffusion based flow-matching (FM) model, with these components trained independently. In this paper, we propose a fully end-to-end (E2E) optimization framework that unifies the training of the speech tokenizer, LLM, FM model, and an additional reward model (RM). Specifically, we first jointly optimize the tokenizer using multi-task objectives derived from reconstruction for FM, next-token prediction for LLM, and multi recognition task for RM. This joint training encourages the discrete speech token space to capture acoustically and semantically salient information that is better tailored to TTS. We then further optimize the LLM using downstream reconstruction and recognition by FM and RM, which reduces inference-time mismatch and steers the LLM toward more preferred generations. Experimental results show that our E2E framework consistently outperforms cascaded baselines. On the Seed-TTS-Eval benchmark, our system achieves a word error rate (WER) of 0.78% and 1.56%, a new SOTA result with a 0.6B-parameter LLM and 0.5B-parameter FM model. These results validate that holistic E2E optimization is critical for improving discrete-token-based TTS systems with a much simpler training pipeline.
