Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel dance-conditioned music generation framework that leverages both unpaired and paired data to address the challenge of limited high-quality synchronized dance-music datasets. By employing pretrained unimodal encoders, beat-guided contrastive pretraining, and a ControlNet-style conditioning module on a text-to-audio diffusion model, the authors enhance the alignment of dance movements with music while maintaining audio quality. Experimental results on the AIST++ dataset show that this approach significantly outperforms existing methods in dance alignment and achieves competitive audio quality, highlighting its effectiveness in real-world applications like choreography support and automatic accompaniment.
Unpaired data can unlock superior dance-music alignment, achieving better results than traditional paired data methods.
Dance-to-music generation is a promising task for applications such as choreography support and automatic accompaniment, where temporal coordination between body movement and sound is essential. In particular, using human joint positions as the motion representation is attractive because they explicitly capture body dynamics while being lightweight, privacy-preserving, and easy to integrate with motion capture and pose-estimation pipelines. A central challenge in this setting, however, is the scarcity of high-quality paired dance-music data, since collecting accurately synchronized pairs is costly and often constrained by copyright and performance rights. This makes it difficult to train end-to-end models solely from paired data. To address this issue, we propose a dance-conditioned music generation framework that efficiently exploits both unpaired and paired data. Our method combines pretrained unimodal encoders for motion and music, beat-guided contrastive pretraining to align their feature spaces, and a ControlNet-style conditioning module on top of a pretrained text-to-audio diffusion model. Experiments on AIST++ demonstrate that the proposed techniques improve both dance-music alignment and audio quality, as confirmed by quantitative and qualitative evaluations. Compared to a state-of-the-art method, our approach achieves superior dance alignment performance and competitive audio quality. Code is available at https://github.com/kmraven/AudioLDM-ControlNet .