Search papers, labs, and topics across Lattice.
This paper introduces Interpretable Motion-Attentive Maps (IMAP) to understand how video diffusion transformers (DiTs) convert motion words into video by localizing motion spatially and temporally. They propose GramCol to generate per-frame saliency maps for text concepts and a motion-feature selection algorithm to obtain IMAP. Experiments demonstrate that IMAP effectively localizes motion and provides interpretable saliency maps for both motion and non-motion concepts, achieving strong performance in motion localization and zero-shot video semantic segmentation.
Uncover the "when and where" behind motion in video diffusion models with a new technique that visualizes motion concepts without gradients or retraining.
Video Diffusion Transformers (DiTs) have been synthesizing high-quality video with high fidelity from given text descriptions involving motion. However, understanding how Video DiTs convert motion words into video remains insufficient. Furthermore, while prior studies on interpretable saliency maps primarily target objects, motion-related behavior in Video DiTs remains largely unexplored. In this paper, we investigate concrete motion features that specify when and which object moves for a given motion concept. First, to spatially localize, we introduce GramCol, which adaptively produces per-frame saliency maps for any text concept, including both motion and non-motion. Second, we propose a motion-feature selection algorithm to obtain an Interpretable Motion-Attentive Map (IMAP) that localizes motion spatially and temporally. Our method discovers concept saliency maps without the need for any gradient calculation or parameter update. Experimentally, our method shows outstanding localization capability on the motion localization task and zero-shot video semantic segmentation, providing interpretable and clearer saliency maps for both motion and non-motion concepts.
