Search papers, labs, and topics across Lattice.
This paper introduces a Face-to-Speech (F2S) framework that synthesizes speech from a static facial image, overcoming the limitations of traditional zero-shot text-to-speech (TTS) systems that require audio prompts. By employing a lightweight Face Adapter and soft-tuning the upper blocks of a face encoder to align with the style space of a frozen StyleTTS 2 model, the authors achieve high-quality voice synthesis that matches or exceeds the naturalness of ground truth audio. Evaluation on the LRS3 corpus demonstrates that the generated speech is not only natural but also consistent with the target speaker's characteristics, and the system exhibits language-agnostic capabilities, producing fluent Spanish speech without retraining.
Synthesizing speech from just a facial image achieves voice quality that rivals traditional methods, with surprising language-agnostic performance.
Zero-shot text-to-speech (TTS) clones a voice from a short audio prompt, but this reliance on reference audio is a barrier when only visual information is available, e.g. for historical figures or video-game characters. In this work, we propose a Face-to-Speech (F2S) framework that predicts a plausible voice from a static facial image. A lightweight Face Adapter, together with soft-tuning of the face encoder's upper blocks, aligns face-recognition features with the style space of a frozen StyleTTS 2 model, kept frozen during training. We evaluate on held-out identities from LRS3, a large-scale audiovisual corpus of English TED-talk videos. The synthesized speech is highly natural (UTMOS 3.7-4.0, matching or exceeding the 3.61 of ground truth), face-to-voice retrieval is consistently above chance, and the generated voice is consistent with the target speaker. Without any retraining, an English-trained adapter also produces fluent Spanish speech, indicating that the face-to-style mapping is largely language-agnostic.
