Search papers, labs, and topics across Lattice.
MotionGrounder, a Diffusion Transformer (DiT) framework, enables multi-object motion transfer in video generation by using a Flow-based Motion Signal (FMS) to provide a stable motion prior. To ensure fine-grained control, the framework incorporates an Object-Caption Alignment Loss (OCAL) that grounds object captions to spatial regions. The method is evaluated using a new Object Grounding Score (OGS) which assesses spatial alignment and semantic consistency, demonstrating superior performance compared to existing methods.
Finally, a diffusion model lets you puppeteer multiple objects in a video with just text prompts, opening the door to complex scene editing.
Motion transfer enables controllable video generation by transferring temporal dynamics from a reference video to synthesize a new video conditioned on a target caption. However, existing Diffusion Transformer (DiT)-based methods are limited to single-object videos, restricting fine-grained control in real-world scenes with multiple objects. In this work, we introduce MotionGrounder, a DiT-based framework that firstly handles motion transfer with multi-object controllability. Our Flow-based Motion Signal (FMS) in MotionGrounder provides a stable motion prior for target video generation, while our Object-Caption Alignment Loss (OCAL) grounds object captions to their corresponding spatial regions. We further propose a new Object Grounding Score (OGS), which jointly evaluates (i) spatial alignment between source video objects and their generated counterparts and (ii) semantic consistency between each generated object and its target caption. Our experiments show that MotionGrounder consistently outperforms recent baselines across quantitative, qualitative, and human evaluations.
