Search papers, labs, and topics across Lattice.
This study investigates the role of preference optimization in generating non-verbal vocalizations (NVs) for text-to-speech (TTS) systems, focusing on preference signals and pair construction. By introducing a novel NV-aware character error rate (NV-CER) metric, the authors enable precise optimization of NV outputs while maintaining the integrity of the TTS algorithm. Experiments demonstrate that specific design choices significantly influence NV realization and lexical accuracy, with results validated through multiple evaluation methods, including human assessments.
Achieving expressive TTS now hinges on effectively optimizing non-verbal vocalizations, with design choices impacting NV fidelity more than previously understood.
Non-verbal vocalizations (NVs), such as laughter, coughs, and sighs, are essential for expressive TTS, but the effectiveness of preference optimization for NV generation remains poorly understood. We systematically study preference optimization for NV-capable TTS, focusing on preference signals, preference-pair construction, and DPO-based optimization objectives. We formulate an NV-aware character error rate (NV-CER) by treating NV tags as distinct output symbols and computing a weighted pinyin-based CER over both verbal and non-verbal content, enabling controllable optimization of NV realization without modifying the underlying optimization algorithm. Experiments on Emilia-NV and the augmented NV-Bench covering 18 NV types reveal how different design choices affect NV realization and lexical fidelity, and establish an effective setup using standard DPO. Objective, LLM-based, and human evaluations provide converging evidence for our findings, offering practical insights into NV-aware post-training for expressive TTS.
