Search papers, labs, and topics across Lattice.
This study investigates the impact of different speech representation methods on the quality of 3D facial animation, focusing on segmental, semantic, and label-based encodings. By evaluating four distinct speech representation families across two facial decoders, the authors reveal that encoding phonetic classes significantly enhances facial animation accuracy, achieving comparable quality across semantic and label-based representations. The introduction of an Audio Visual Text-to-Speech (AVTTS) pipeline demonstrates the practical application of discrete representations in synchronizing speech and facial motion effectively.
Encoding phonetic classes can dramatically improve the accuracy of 3D facial animation, unlocking new possibilities for realistic speech-driven avatars.
The choice of speech representation is critical in speech-driven 3D facial animation. Representations differ in what they encode: SSL features emphasize segmental and semantic cues, neural codecs yield latents optimized for acoustic reconstruction, and ASR-style objectives produce label-based spaces. We evaluate four speech representation families for 3D facial synthesis, comparing their facial reconstruction quality across two facial decoders using objective metrics and a perceptual evaluation. We additionally conduct probing analyses that relate tokenized representations to phonetic units and to articulatory deformations. We found that encoding phonetic classes is beneficial for accurate facial animation prediction on both semantic and label-based representations with comparable facial animation quality. From the latter, we introduce an Audio Visual Text-to-Speech (AVTTS) pipeline that leverages, as a shared space, discrete representations to decode speech and 3D facial motion.
