Search papers, labs, and topics across Lattice.
This paper introduces Phoenix TTS, a novel framework that integrates representation learning with generative acoustic modeling to enhance zero-shot text-to-speech synthesis and voice conversion. By optimizing a speech tokenizer to reconstruct self-supervised features while receiving direct supervision from a Flow Matching loss, the system effectively preserves semantic richness and aligns with the continuous feature space required for high-fidelity acoustic generation. Evaluations demonstrate that Phoenix TTS achieves superior speech intelligibility and speaker similarity, outperforming existing large-scale models while enabling seamless adaptation for voice conversion tasks.
Achieving speech intelligibility that consistently surpasses ground-truth recordings, Phoenix TTS redefines the boundaries of zero-shot TTS and voice conversion.
In current zero-shot text-to-speech systems, conventional semantic tokenizers are typically optimized using supervised automatic speech recognition or self-supervised learning objectives. However, due to the inherent nature of speech, semantic and acoustic information cannot be completely decoupled, and ASR-based tokenizers discard acoustic details to focus on linguistic content; models relying on them usually struggle to achieve optimal speaker similarity. Furthermore, these tokenizers are optimized independently and lack direct supervision from downstream acoustic generation tasks. This isolated training creates a feature gap between the extracted discrete tokens and the continuous space required by acoustic models, fundamentally bottlenecking the upper bound of synthesis quality. To bridge this gap, we propose Phoenix TTS, a unified framework that tightly couples representation learning with generative acoustic modeling. Specifically, our speech tokenizer is optimized to reconstruct self-supervised features to maintain semantic richness, while simultaneously receiving direct supervision from a Flow Matching training loss. Through this joint training paradigm, the extracted discrete tokens successfully preserve essential semantic information and natively align with the feature space of the downstream Flow Matching model. Comprehensive evaluations highlight the efficiency and effectiveness of Phoenix TTS. Trained on 110K hours of data, the system achieves excellent speech intelligibility, yielding WER that consistently falls below that of ground-truth recordings. Simultaneously, it maintains robust zero-shot speaker similarity, rivaling or outperforming several prominent large-scale baselines. Furthermore, as an advantageous byproduct of this unified training, the learned tokenizer can be seamlessly adapted to zero-shot voice conversion tasks without requiring task-specific fine-tuning.
