Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
T5Gemma-TTS, a 4B parameter encoder-decoder codec language model, was developed to improve zero-shot voice cloning by maintaining persistent text conditioning through cross-attention at every decoder layer, addressing limitations of decoder-only architectures. The model incorporates Progress-Monitoring Rotary Position Embedding (PM-RoPE) to enhance duration control, trained on 170k hours of multilingual speech data. Results show statistically significant speaker-similarity gains in Japanese compared to XTTSv2 and strong zero-shot generalization to Korean, demonstrating the effectiveness of the architecture and training methodology.
Cross-attention and progress monitoring make T5Gemma-TTS a zero-shot voice cloning model that not only beats existing systems in speaker similarity, but also generalizes to unseen languages.
Autoregressive neural codec language models have shown strong zero-shot voice cloning ability, but decoder-only architectures treat input text as a prefix that competes with the growing audio sequence for positional capacity, weakening text conditioning over long utterances. We present T5Gemma-TTS, an encoder-decoder codec language model that maintains persistent text conditioning by routing bidirectional text representations through cross-attention at every decoder layer. Built on the T5Gemma pretrained encoder-decoder backbone (2B encoder + 2B decoder; 4B parameters), it inherits rich linguistic knowledge without phoneme conversion and processes text directly at the subword level. To improve duration control, we introduce Progress-Monitoring Rotary Position Embedding (PM-RoPE) in all 26 cross-attention layers, injecting normalized progress signals that help the decoder track target speech length. Trained on 170,000 hours of multilingual speech in English, Chinese, and Japanese, T5Gemma-TTS achieves a statistically significant speaker-similarity gain on Japanese over XTTSv2 (0.677 vs. 0.622; non-overlapping 95% confidence intervals) and the highest numerical Korean speaker similarity (0.747) despite Korean not being included in training, although this margin over XTTSv2 (0.741) is not statistically conclusive. It also attains the lowest numerical Japanese character error rate among five baselines (0.126), though this ranking should be interpreted cautiously because of partial confidence-interval overlap with Kokoro. English results on LibriSpeech should be viewed as an upper-bound estimate because LibriHeavy is a superset of LibriSpeech. Using the same checkpoint, disabling PM-RoPE at inference causes near-complete synthesis failure: CER degrades from 0.129 to 0.982 and duration accuracy drops from 79% to 46%. Code and weights are available at https://github.com/Aratako/T5Gemma-TTS.