Search papers, labs, and topics across Lattice.
This paper introduces VoiceTTA, a novel reinforcement learning-based test-time adaptation method designed to enhance zero-shot text-to-speech (TTS) systems' ability to imitate unseen speaking styles. By incorporating two style rewards鈥攃oefficient-of-variation differences for pitch and energy鈥攁longside speaker similarity and intelligibility metrics, VoiceTTA optimizes learnable prefixes during inference to significantly improve performance on challenging speech prompts. Experimental results show that VoiceTTA outperforms existing state-of-the-art methods, highlighting its effectiveness in personalizing TTS without the need for extensive fine-tuning datasets.
VoiceTTA transforms zero-shot TTS by enabling real-time adaptation to unseen speaking styles, achieving remarkable improvements in performance on challenging prompts.
Recently, zero-shot text-to-speech (TTS) has enabled high-fidelity and expressive speech synthesis, but it often fails to imitate unseen speaking styles from uncommon scenarios (e.g., crosstalk, dialects). Moreover, fine-tuning pretrained models requires large, high-quality datasets, limiting rapid personalization. We propose VoiceTTA, a reinforcement learning-based test-time adaptation (TTA) method that improves voice imitation of pretrained zero-shot TTS models. VoiceTTA introduces two style rewards based on coefficient-of-variation differences of F0 and energy, combined with speaker similarity and intelligibility (WER from a pretrained Whisper model), and optimizes learnable prefixes via group relative preference optimization (GRPO) in a flow matching-based model at inference time. Extensive experiments demonstrate substantial improvements on uncommon speech prompts, outperforming state-of-the-art baselines. Audio samples are available at https://voicetta.pages.dev/
