Search papers, labs, and topics across Lattice.
This paper introduces an Iterative Self-Learning (ISL) framework for expressive text-to-speech (TTS) synthesis that addresses the challenge of acquiring expressive labels at scale. By employing a classifier-free method called Invert-Classify, the framework iteratively pseudo-labels unlabeled speech and retrains on both labeled and pseudo-labeled data, leading to improved label quality and synthesis. The results show that ISL significantly enhances expressive label adherence and synthesis quality, particularly in low-resource scenarios, outperforming traditional single-pass methods and nearing fully supervised performance.
Iterative refinement in TTS can bridge the gap between low-resource data conditions and high-quality expressive synthesis, achieving near-supervised performance with minimal labeled data.
Expressive text-to-speech (TTS) systems that use explicit conditioning labels provide direct and interpretable control over expressive attributes, in contrast to reference-based or prompting-based approaches, but require labeled data. Obtaining these labels at scale is costly and time-consuming, yet no prior semi-supervised framework addresses this specific bottleneck. Existing semi-supervised TTS methods instead target scarcity of paired speech-text data or transcriptions. To address the scarcity of expressive labels, we propose an Iterative Self-Learning (ISL) framework for expressive TTS, built on Invert-Classify, a classifier-free method that recovers discrete expressive labels by inverting a frozen generative model. The framework iteratively pseudo-labels unlabeled speech using the current model, retrains on the combined labeled and pseudo-labeled data, and repeats, progressively refining label quality and synthesis. We validate on two expressive tasks, word-level prominence and utterance-level emotion, across multiple low-resource data splits. We find that iterative refinement can improve pseudo-label accuracy over single-pass baselines. Furthermore, we observe that these improvements in pseudo-labeling of expressivity translate to gains in expressive label adherence and synthesis quality, confirmed by objective metrics and human listening tests. In the most data-scarce conditions, ISL-trained models outperform single-pass pseudo-labeling and further approach fully supervised performance, demonstrating that gradient-based ISL is an effective solution to expressive label scarcity in low-resource TTS.
