Search papers, labs, and topics across Lattice.
This paper introduces RTFree-F5, a novel approach to text-to-speech (TTS) synthesis that eliminates the reliance on reference transcripts by utilizing continuous self-supervised speech representations. This method addresses the brittleness of existing models like F5-TTS, particularly for accented and dysarthric speakers, by mapping speech features directly into the text-conditioning space. The results show a significant reduction in word error rate (WER) from 24.6% to 10.4% on dysarthric speech, outperforming traditional methods that use ground-truth transcripts while enhancing naturalness and maintaining competitive performance on standard benchmarks.
RTFree-F5 achieves a remarkable 10.4% WER on dysarthric speech without needing any reference transcripts, surpassing even ground-truth baselines.
Recent flow-matching text-to-speech (TTS) models, such as F5-TTS, rely on a reference transcript at inference time, obtained from an external ASR system. This dependency makes zero-shot TTS brittle for accented or dysarthric speakers, precisely the scenarios where it is most needed. Moreover, we find that text-based reference conditioning can propagate atypical acoustic patterns from atypical speech into synthesis, even when ground-truth transcripts are available. To address this, we propose RTFree-F5, which replaces the reference transcript with continuous self-supervised speech representations mapped into F5-TTS's text-conditioning space via a lightweight adapter, while reusing the pretrained checkpoint. On dysarthric speech, RTFree-F5 reduces WER from 24.6% to 10.4%, surpassing even the ground-truth reference transcript baselines, while improving naturalness and remaining competitive on standard benchmarks without requiring any reference transcript.
