Search papers, labs, and topics across Lattice.
ViDS is a novel Video Diffusion Shader that utilizes 3D face tracking to create expressive and identity-preserving portrait animations. By reconstructing a 3D Morphable Model (3DMM) mesh from a reference image and animating it with parameters derived from a driving video, the method achieves superior expression control and continuity in generated animations. The autoregressive diffusion sampling process allows for extended generation beyond the model's native window, resulting in more detailed and consistent outputs compared to previous landmark-based approaches.
More accurate 3D face tracking in ViDS enables unprecedented control over expression and pose in portrait animations, setting a new standard for identity preservation.
We introduce ViDS, a Video Diffusion Shader that leverages 3D face tracking for expressive and identity-preserving portrait animation. We first reconstruct the identity-specific 3DMM mesh from the reference image, and then animate it using expression and pose parameters from a driving video. Leveraging dense geometric cues from 3DMM normal maps, we employ a video diffusion model as a neural shader to synthesize lifelike portrait animations while preserving the appearance and identity of the reference image. We find that more accurate 3DMM tracking enables finer-grained expression control. We also introduce an autoregressive diffusion sampling process that extends generation beyond the model's native window while reducing discontinuities between adjacent clips. Compared with prior diffusion-based approaches for portrait animation that rely on landmark-based conditioning or implicit motion latents, our method achieves more detailed and consistent expression and pose control while faithfully preserving identity and appearance. Detailed ablation studies validate the effectiveness of our design choices. Project page: https://fusheng-ji.github.io/ViDS/
