Search papers, labs, and topics across Lattice.
This paper introduces a novel system for generating spatially-aware 3D facial animations of two co-located individuals engaged in conversation, driven by a mixed audio stream. The system employs a dual-stream architecture with speaker role embeddings and cross-attention to disentangle audio and model inter-speaker interaction, along with a novel eye gaze loss to encourage natural eye contact. The approach is trained on a newly curated dataset of over 2 million dyadic pairs extracted from in-the-wild videos, and demonstrates superior performance in realism and interaction coherence compared to existing baselines.
Finally, realistic 3D avatars can maintain natural eye contact and spatial awareness during conversations, moving beyond disembodied "talking heads."
We tackle the challenging task of generating complete 3D facial animations for two interacting, co-located participants from a mixed audio stream. While existing methods often produce disembodied "talking heads" akin to a video conference call, our work is the first to explicitly model the dynamic 3D spatial relationship -- including relative position, orientation, and mutual gaze -- that is crucial for realistic in-person dialogues. Our system synthesizes the full performance of both individuals, including precise lip-sync, and uniquely allows their relative head poses to be controlled via textual descriptions. To achieve this, we propose a dual-stream architecture where each stream is responsible for one participant's output. We employ speaker's role embeddings and inter-speaker cross-attention mechanisms designed to disentangle the mixed audio and model the interaction. Furthermore, we introduce a novel eye gaze loss to promote natural, mutual eye contact. To power our data-hungry approach, we introduce a novel pipeline to curate a large-scale conversational dataset consisting of over 2 million dyadic pairs from in-the-wild videos. Our method generates fluid, controllable, and spatially aware dyadic animations suitable for immersive applications in VR and telepresence, significantly outperforming existing baselines in perceived realism and interaction coherence.
