Search papers, labs, and topics across Lattice.
This study investigates the alignment of predictor-based reinforcement learning (RL) with human perception in codec-based text-to-speech (TTS) models, focusing on subjective rewards for speaking style, naturalness, likability, and arousal. By employing Group Relative Policy Optimization (GRPO) and introducing a character error rate (CER) zone constraint, the authors reveal that different subjective predictors enhance their respective metrics but are not interchangeable. The findings indicate that while GRPO can effectively optimize speech quality, it does not consistently outperform traditional Best-of-$N$ reranking methods, emphasizing the need for careful reward selection in speech generation tasks.
Subjective rewards in speech generation are not interchangeable, revealing critical nuances in how RL aligns with human listener preferences.
Codec-based text-to-speech (TTS) models make language-model post-training applicable to speech generation, but it remains unclear when learned perceptual predictors can serve as reinforcement learning rewards without losing alignment with human listeners. We study this question with Group Relative Policy Optimization (GRPO) using learned rewards for anime-like speaking style, naturalness, likability, and arousal. To prevent perceptual rewards from being optimized through transcript drift, we introduce a character error rate (CER) zone constraint and compare policy optimization with Best-of-$N$ reranking under the same reward gate. Across single-reward runs, each reward primarily improves its own target metric, showing that subjective predictors are not interchangeable quality surrogates. Multi-rater A/B tests further show uneven human transfer, while a reward-gap analysis separates average transfer from within-axis calibration: signed reward gaps significantly predict listener choices in the pooled analysis, whereas residual CER gaps do not, but per-axis calibration remains heterogeneous. Best-of-8 is a strong human-level baseline and is not clearly worse than GRPO perceptually, suggesting that GRPO should be viewed as amortizing reward-selected behavior into the policy rather than uniformly outperforming reranking. These results support analyzing subjective speech rewards as predictor-axis-base tuples and provide practical diagnostics for selecting rewards before multi-reward speech post-training.
