Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This research introduces a deployable system for speech-driven 3D facial animation in Unreal Engine, utilizing ARKit-compatible representations to enhance production compatibility. By constructing the 3DMEAD-ARKit dataset and retraining existing models like FaceDiffuser and ProbTalk3D-X, the authors achieve stochastic and emotion-controllable animations. Comparative evaluations against commercial tools reveal significant insights into the effectiveness of academic versus industry solutions in this domain.
Bridging the gap between academic research and production-ready tools, this system enables realistic speech-driven 3D facial animations directly in Unreal Engine.
Speech-driven 3D facial animation research has shown promising results, but most methods rely on representations that are not compatible with production pipelines. In this work, we present a deployable system that bridges this gap by enabling speech-driven 3D facial animation directly in Unreal Engine (UE) using ARKit-compatible representations. We construct 3DMEAD-ARKit dataset by converting the MEAD corpus into blendshape sequences using MediaPipe, and retrain FaceDiffuser and ProbTalk3D-X to generate stochastic and emotion controllable animations. We further develop a modular UE plugin with a Python backend that supports model selection, and parameter control. We compare the results to two existing commercial tools: Epic Games' MetaHuman speech-driven animator and Nvidia Audio2Face with a perceptual user study. The results highlight the importance of comparisons among academic and commercial pipelines. We recommend watching the supplementary video. We also plan to do live demonstrations of our work at Siggraph 2026 conference.