Search papers, labs, and topics across Lattice.
This paper introduces a scene-adaptive framework for human image animation that enables trajectory-controllable motion and camera movement within a 3D environment. By employing a ground-adaptive 3D motion retargeting approach and a viewpoint-adaptive latent fusion mechanism, the authors effectively manage user-defined motion trajectories while accommodating changes in ground elevation and camera angles. Experimental results on benchmark datasets show significant improvements over existing methods in video generation metrics, highlighting the framework's capability to synthesize realistic animations in dynamic scenes.
Achieving precise control over human motion and camera trajectories in 3D scene animation could redefine the realism and interactivity of video generation.
Human image animation, which aims to generate a video of a reference subject following a provided action sequence, has received increasing research interest. With the development of diffusion-based/flow-based video foundation models, existing animation works have began to upgrade the guidance information from 2D skeleton/pose to 3D modeling conditions. Despite achieving reasonable results, these approaches face challenges in synthesizing trajectory-controllable human motion within natural scene under changed camera views. In this work, we present a scene-adaptive human image animation framework that controls both human motion and camera trajectories within a reconstructed 3D environment for video generation. To achieve this, we first develop a ground-adaptive 3D motion retargeting approach to enable user-friendly motion trajectory control adapting to the changes of elevations of ground and orientations automatically. Then we design a viewpoint-adaptive latent fusion mechanism to inject point-cloud geometric priors through scene-visibility masking into the generative process, providing precise guidance of viewpoint changes under camera control. Experiments on two standard human image animation benchmark datasets demonstrate remarkable improvements of our method over the state of the arts in related video generation metics. Project page: https://robinhood256100.github.io/web-disp
