Search papers, labs, and topics across Lattice.
This paper introduces QWERTY, a training-free framework that enhances motion control in pretrained video diffusion transformers (DiTs) by utilizing user-defined object warping and optical flow. By manipulating the 3D full attention of DiTs through query warping, the method effectively guides the diffusion trajectory, allowing for flexible and stable motion control without the need for extensive prompt engineering or fine-tuning. Experimental results demonstrate that QWERTY outperforms existing training-free methods and achieves performance on par with fine-tuning approaches, marking a significant advancement in motion control for video generation tasks.
Achieving high-fidelity motion control in video diffusion transformers without any training or extensive prompt engineering could revolutionize how we generate dynamic video content.
Video diffusion transformers (DiTs) generate high-fidelity and temporally coherent videos, yet motion control remains implicit, primarily relying on text prompts. As a result, achieving desired motion often requires extensive prompt engineering and repeated resampling. While fine-tuning models with additional spatial prompts (e.g., bounding boxes or point trajectories) enables explicit control, it demands substantial data curation and computation, and may compromise the generative capabilities of pretrained models. Consequently, training-free motion control using such spatial prompts has been explored in U-Net-based video diffusion models, but remains largely unexplored for DiTs. We introduce QWERTY, a training-free framework that enables flexible motion control in pretrained image-to-video DiTs via user-defined object warping and optical flow. We carefully manipulate the 3D full attention of DiTs by warping the frame-invariant semantic subspace of queries. We find that the noise predicted by the query-warped DiT naturally guides the diffusion trajectory toward the desired motion, and further show that leveraging this noise as self-guidance for latent optimization improves control stability and visual quality. Experiments show that QWERTY achieves the most effective motion control among existing training-free approaches on a recent image-to-video DiT, with performance comparable to fine-tuning-based methods.
