Search papers, labs, and topics across Lattice.
This paper introduces CIPE-Dance, the largest dataset for dance video generation, comprising 300k high-quality clips with choreography-informed text annotations, addressing the critical lack of large-scale datasets in this domain. It also presents OmniDance, a novel framework that effectively integrates music into a TI2V foundation model while preserving controllability and visual fidelity through a depth-aware specialization architecture and a modality-specialized time-dependent CFG strategy. Experimental results show that OmniDance achieves state-of-the-art performance in generating dance videos aligned with music, demonstrating its robust multimodal integration capabilities.
CIPE-Dance is a game-changer, providing the largest dataset for dance video generation and enabling OmniDance to set new benchmarks in multimodal video synthesis.
Music-driven dance video generation aims to synthesize expressive human motion that is temporally aligned with music while maintaining high visual fidelity. Despite recent progress, existing methods still face two key limitations: the lack of large-scale, high-quality dance video datasets, and the absence of principled frameworks for integrating music as a complementary conditioning signal into Video Generation Foundation Models. To address these limitations, we introduce CIPE-Dance, a large-scale Internet-sourced dance video dataset with choreography-informed text annotations, constructed via a progressive expert pipeline. To the best of our knowledge, CIPE-Dance is the largest dataset for dance video generation to date, comprising 300k high-quality clips over 400 hours and covering diverse dancers, environments, and dance genres. We further propose OmniDance, a framework-level recipe for integrating music into a TI2V foundation model without sacrificing its original controllability or visual fidelity. Motivated by the complementary roles of text as low-frequency semantics and music as high-frequency temporal dynamics, OmniDance co-designs a depth-aware specialization architecture, an anchored easy-to-hard curriculum learning strategy, and a modality-specialized time-dependent CFG strategy, enabling unified TI2V, MI2V, and MTI2V generation. Extensive experiments on CIPE-Dance demonstrate that OmniDance achieves state-of-the-art performance across all three tasks and exhibits robust multimodal integration capability. Project is available at https://github.com/AMAP-ML/OmniDance.
DAMO
Tsinghua AI