Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MVFGA, a novel pipeline for creating photorealistic upper-body avatars that accurately animate facial expressions and hand gestures by separately modeling the face and hands and fusing them with a parametric upper-body mesh. The approach addresses the limitations of existing methods that either neglect hand gestures or compromise facial fidelity, thus enhancing the realism of human communication in digital avatars. Experimental results demonstrate that MVFGA significantly outperforms baseline methods in generating high-fidelity animations, supported by the introduction of a new motion capture dataset, MVFGA-MoCap, which includes diverse hand gestures and facial expressions.
Achieving unprecedented realism in upper-body avatar animations by accurately capturing fine-grained facial expressions and hand gestures could revolutionize virtual communication.
Creating photorealistic 3D human avatars with realistic upper-body motion remains challenging. Existing approaches either focus on the head and overlook hand gestures, or reconstruct the full body but fail to preserve fine-grained facial fidelity and hand pose accuracy. As a result, current methods struggle to capture the subtle dynamics of facial expressions and hand gestures that are crucial for natural human communication. While methods based on full-body parametric models enable avatar reconstruction from monocular or multi-view inputs, they often lack accurate facial animation and detailed hand articulation. To address these limitations, we propose MVFGA, a novel multi-view-consistent pipeline for generating realistic upper-body avatars. Our approach models the face and hands separately and fuses them with a parametric upper-body mesh model, enabling the capture of fine-grained facial expressions and hand poses for accurate upper-body avatar reconstruction. We then splat 3D Gaussians onto the obtained mesh, enabling high-quality rendering of dynamic avatars from novel viewpoints. Furthermore, we introduce MVFGA-MoCap, a multi-view upper-body motion capture dataset featuring controlled facial expression sequences, diverse hand gestures, and free-form communication. Experiments show that MVFGA generates visually realistic avatars with high-fidelity facial expressions and hand motions, outperforming baselines for upper-body avatar animation. Project page: https://dfki-av.github.io/MVFGA/