Search papers, labs, and topics across Lattice.
This paper introduces a motion-adaptive temporal attention mechanism for lightweight video generation using frozen Stable Diffusion models. The method dynamically adjusts temporal attention receptive fields based on estimated motion content, using local attention for high-motion sequences and global attention for low-motion sequences. By injecting lightweight temporal attention modules into the UNet and using temporally correlated noise initialization, the system achieves competitive video generation results with only 2.9% additional trainable parameters, demonstrating that the standard denoising objective provides sufficient temporal regularization.
Achieve competitive video generation with Stable Diffusion using only 2.9% additional parameters by adapting temporal attention based on motion content, outperforming methods with explicit temporal consistency losses.
We present a motion-adaptive temporal attention mechanism for parameter-efficient video generation built upon frozen Stable Diffusion models. Rather than treating all video content uniformly, our method dynamically adjusts temporal attention receptive fields based on estimated motion content: high-motion sequences attend locally across frames to preserve rapidly changing details, while low-motion sequences attend globally to enforce scene consistency. We inject lightweight temporal attention modules into all UNet transformer blocks via a cascaded strategy -- global attention in down-sampling and middle blocks for semantic stabilization, motion-adaptive attention in up-sampling blocks for fine-grained refinement. Combined with temporally correlated noise initialization and motion-aware gating, the system adds only 25.8M trainable parameters (2.9\% of the base UNet) while achieving competitive results on WebVid validation when trained on 100K videos. We demonstrate that the standard denoising objective alone provides sufficient implicit temporal regularization, outperforming approaches that add explicit temporal consistency losses. Our ablation studies reveal a clear trade-off between noise correlation and motion amplitude, providing a practical inference-time control for diverse generation behaviors.
