Search papers, labs, and topics across Lattice.
This paper introduces FacePlex, a novel framework for full-duplex joint speech and facial motion generation that operates in real-time. By employing Rolling Flow Matching and Rolling Cross-Attention, FacePlex allows for the simultaneous generation of speech and synchronized facial animations, overcoming the limitations of existing systems that either focus solely on speech or rely on pre-existing audio. Experimental results demonstrate that FacePlex significantly improves lip-sync quality and motion fidelity compared to traditional audio-driven models, making it a substantial advancement in conversational avatar technology.
FacePlex achieves real-time, synchronized speech and facial motion generation, outperforming existing models in lip-sync quality and motion fidelity.
Natural face-to-face conversation requires real-time speech generation together with synchronized facial motion. Existing systems only partially address this problem: speech-only full-duplex models can generate speech in real time but do not produce facial motion, while audio-driven facial motion models animate a face from already available audio rather than jointly generating speech and motion online. To bridge this gap, we first formalize full-duplex joint speech-facial motion generation, where speech tokens and facial motion tokens are produced together every step. Building on this formulation, we propose FacePlex, a unified streaming framework with two key components. First, Rolling Flow Matching adapts flow matching to online motion generation by committing new motion frames at each streaming step. Second, Rolling Cross-Attention couples the streaming audio queue with the motion queue, allowing speech and facial motion to condition each other as generation progresses. Through extensive experiments, ablation studies, and a user study, we show that FacePlex enables full-duplex joint speech-facial motion generation under online streaming constraints, while achieving stronger lip-sync quality and motion fidelity than audio-driven facial motion baselines.
