Search papers, labs, and topics across Lattice.
This paper introduces a framework for generating realistic lip movements in humanoid robots by constructing a 3D dynamic viseme library based on ARKit standards and modeling coarticulation effects using initial-final decoupling and energy modulation. The approach addresses the challenge of achieving natural non-verbal communication in human-robot interaction by creating coherent lip trajectories. Experimental results, validated with PCC and MAJ metrics, demonstrate the efficiency and accuracy of the proposed architecture when retargeting high-dimensional lip motion to a 14-DOF lip actuation system.
Achieve realistic and efficient speech-driven lip synchronization for humanoid robots by decoupling initial and final phoneme articulations and modulating energy.
Realistic lip synchronization is essential for the natural human-robot non-verbal interaction of humanoid robots. Motivated by this need, this paper presents a lip motion generation framework based on 3D dynamic viseme and coarticulation modeling. By analyzing Chinese pronunciation theory, a 3D dynamic viseme library is constructed based on the ARKit standard, which offers coherent prior trajectories of lips. To resolve motion conflicts within continuous speech streams, a coarticulation mechanism is developed by incorporating initial-final (Shengmu-Yunmu) decoupling and energy modulation. After developing a strategy to retarget high-dimensional spatial lip motion to a 14-DOF lip actuation system of a humanoid head platform, the efficiency and accuracy of the proposed architecture is experimentally validated and demonstrated with quantitative ablation experiments using the metrics of the Pearson Correlation Coefficient (PCC) and the Mean Absolute Jerk (MAJ). This research offers a lightweight, efficient, and highly practical paradigm for the speech-driven lip motion generation of humanoid robots. The 3D dynamic viseme library and real-world deployment videos are available at {https://github.com/yuesheng21/Phoneme-to-Lip-14DOF}
