Search papers, labs, and topics across Lattice.
This paper introduces GROW, a novel group-relative advantage-weighted on-policy reinforcement learning method tailored for autoregressive-diffusion text-to-speech (TTS) models, addressing the challenges posed by deterministic ODE sampling. By sampling groups of on-policy utterances and standardizing rewards for intelligibility and speaker similarity, GROW effectively reweights the flow-matching regression while incorporating a Wasserstein-2 velocity penalty to stabilize the model against a frozen pretrained reference. The method demonstrates significant improvements in word error rate (WER) and speaker similarity, achieving a WER reduction from 2.016 to 1.558 and an increase in speaker similarity from 0.676 to 0.715, all while maintaining training efficiency and model performance.
GROW achieves a 22.7% reduction in word error rate while accelerating training by 2.9x, redefining efficiency in TTS reinforcement learning.
Reinforcement learning for flow-matching text-to-speech is complicated by deterministic ODE sampling: trajectory-level policy-gradient methods typically convert the ODE into an SDE and track per-step likelihood ratios, introducing stochastic perturbations and substantial overhead. We propose GROW, a group-relative advantage-weighted on-policy RL method that acts directly on the standard flow-matching objective. For each prompt, GROW samples a group of on-policy utterances, separately standardizes intelligibility and speaker-similarity rewards within the group, and combines them to reweight flow-matching regression. A Wasserstein-2 velocity penalty anchors the updated model to a frozen pretrained reference. A group-mean reward baseline is introduced to convert reward weighting into advantage weighting. For strong pretrained TTS models with concentrated rewards, positive exponential weighting is dominated by reward-agnostic self-imitation, whereas a zero-mean signed advantage preserves effective within-group credit assignment. Instantiated on DiTAR and evaluated on LibriSpeech and Seed-TTS EN/ZH, GROW reduces average WER from 2.016 to 1.558 and raises speaker similarity from 0.676 to 0.715 while keeping UTMOS. With 10-NFE training rollouts and 32-NFE evaluation, GROW retains comparable performance while training 2.9x faster than 32-NFE DiTAR-GRPO. We will open-source complete GROW codes, faithful DiTAR reproduction, and all model checkpoints.
