Search papers, labs, and topics across Lattice.
This study investigates the process of phoneme addition during fine-tuning in speech synthesis, particularly when the target corpus includes unseen phonemes. Using both a simulation setup with LLM-generated phoneme-controlled corpora and a real-speech cross-lingual transfer from English to Japanese, the authors found that fine-tuning improved naturalness compared to training from scratch but required substantial data to achieve similar performance in phoneme error rates (PER) for new phonemes. The key finding reveals that while pre-training enhances the naturalness of generated speech, it provides limited advantages for the addition of new phonemes during fine-tuning.
Fine-tuning for phoneme addition may enhance naturalness but demands as much data as training from scratch to achieve comparable performance on new phonemes.
Transfer learning is widely used for low-resource text-to-speech. When the target corpus contains phonemes unseen in pre-training, the model must expand its phoneme inventory during fine-tuning; we call the process "phoneme addition." However, it remains unclear whether the pre-trained ability to generate seen phonemes contributes to this process. This study investigates phoneme addition in two settings: (1) a simulation setup using LLM-generated phoneme-controlled corpora that enables investigation without considering confounding factors, and (2) a real-speech cross-lingual transfer setup (English to Japanese) to validate whether the findings hold in practice. Experiments in both settings showed that while fine-tuning achieved higher naturalness than training from scratch, it required as much or more data to achieve comparable PER for new phonemes. These results indicate that pre-training mainly contributes to naturalness improvement, but offers limited benefit for phoneme addition.
