Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of extracting style vectors from frozen text-to-speech (TTS) models, which typically do not provide a reference encoder for users to generate their own voice styles. By employing gradient descent to optimize only the style vector while keeping the model weights fixed, the authors successfully invert the synthesis pipeline, allowing for the generation of personalized voice styles without requiring transcripts or audio alignment. The method significantly improves voice similarity metrics, with ECAPA-TDNN scores increasing from 0.132 to 0.413 and ResNet scores from 0.099 to 0.401 across 154 speakers, demonstrating a substantial enhancement in voice recovery accuracy compared to preset styles.
Recovering personalized voice styles from frozen TTS models can boost similarity scores by over 200%, revolutionizing user customization in speech synthesis.
Some text-to-speech systems ship a synthesis model and preset style vectors but not the reference encoder that turns audio into such a vector. The model still accepts a style vector; a user with a voice of their own cannot produce one. We solve for that input directly, inverting the released pipeline by gradient descent: every weight stays frozen and only the style vector is optimized, against time-pooled WavLM statistics of one recording. Because the objective discards the time axis, the synthesized text may differ from the recording, so no transcript and no alignment are needed. On 154 speakers from two corpora, ECAPA-TDNN similarity rises from 0.132 to 0.413 and ResNet from 0.099 to 0.401, improving for every speaker; a verifier at its equal-error point accepts 53% of the recovered voices as the target, against 1% for the presets they start from.