Search papers, labs, and topics across Lattice.
The paper introduces UniTAF, a modular framework that unifies Text-to-Speech (TTS) and Audio-to-Face (A2F) models to facilitate feature transfer and improve audio-visual consistency in generated content. The framework explores reusing intermediate TTS representations for joint modeling, extending emotion control from TTS to the combined model. The study validates the feasibility of this unified approach and provides practical engineering insights for co-designing speech and facial expressions, focusing on system design rather than generation quality.
Unifying text-to-speech and audio-to-face models unlocks shared feature representations, paving the way for more consistent and expressive virtual characters.
This work considers merging two independent models, TTS and A2F, into a unified model to enable internal feature transfer, thereby improving the consistency between audio and facial expressions generated from text. We also discuss the extension of the emotion control mechanism from TTS to the joint model. This work does not aim to showcase generation quality; instead, from a system design perspective, it validates the feasibility of reusing intermediate representations from TTS for joint modeling of speech and facial expressions, and provides engineering practice references for subsequent speech expression co-design. The project code has been open source at: https://github.com/GoldenFishes/UniTAF
