Search papers, labs, and topics across Lattice.
This paper introduces the Open Screen Soundtrack Library version 2 (OSSL-v2), a reproducible dataset of 34,343 video clips sourced from public-domain films, addressing the reproducibility issues associated with traditional video-to-music generation datasets. The authors investigate the role of dialogue as a conditioning signal for generating music that aligns with video content, enhancing existing models by incorporating dialogue tracks into the video cross-attention mechanism. Results demonstrate that their approach outperforms state-of-the-art baselines when evaluated on both public-domain and commercial films, highlighting the effectiveness of dialogue in improving video-to-music generation.
Leveraging dialogue as a conditioning signal significantly enhances video-to-music generation, outperforming existing models with a novel dataset that ensures reproducibility.
Video-to-music generation has drawn growing interest for its role in conveying the emotion of visual media, including film. Progress in the field, however, is hampered by a reproducibility gap: models are often trained on crawled corpora referenced through YouTube URLs that may be deleted, with the underlying data often difficult and time-consuming to retrieve. To address this, we introduce the Open Screen Soundtrack Library version 2 (OSSL-v2), a self-hosted corpus of 34,343 video clips totaling 246.4 hours, sourced from public-domain films. Unlike crawled corpora, OSSL-v2 is reproducible (i.e., not subject to link rot) and copyright-conscious, yet still large enough to train functional video-to-music models. We then use this film-domain corpus to study dialogue as a conditioning signal for video-to-music generation, motivated by the close temporal coupling between film music and on-screen speech. Specifically, we augment existing models'video cross-attention with a time axis and modulate it frame-by-frame with the dialogue track. Evaluated on both public-domain and commercial films, our approach shows improvement over the state-of-the-art baselines. The dataset is available at https://huggingface.co/datasets/McAuley-Lab/OSSL-v2.
