Search papers, labs, and topics across Lattice.
The VTMR framework introduces a two-stage approach for video-to-music recommendation, first aligning video and music signals in a joint representation space for efficient semantic retrieval, followed by a temporal reranking process that enhances the matching of sequences. This method significantly improves retrieval metrics, achieving a R@10 score of 18.3 and a Median Rank of 46, outperforming both a strong baseline and a generative model in music quality. The results underscore the importance of both semantic alignment and temporal correspondence in multimodal recommendation systems.
Temporal alignment in video-to-music recommendations can boost retrieval performance by over 20%, revealing the critical role of sequence matching in multimodal tasks.
We present VTMR, a two-stage framework for Video-To-Music Recommendation. In Stage~1, VTMR aligns comprehensive video and music signals in a joint audio-visual-text representation space and efficiently retrieves semantically compatible candidates using coarse global embeddings. In Stage~2, it reranks the retrieved candidates by attending to the temporal sequences of both video and music, thereby capturing fine-grained temporal correspondence. Evaluated on the video-to-music recommendation task, the multimodal retrieval stage improves R@10 from 14.2 to 15.9 and Median Rank from 75 to 58 over the strongest baseline; the temporal reranker further boosts R@10 to 18.3 and Median Rank to 46, demonstrating complementary gains from richer query encoding and temporal alignment. A human preference study confirms that VTMR is on par with a commercial baseline in overall preference, while outperforming a generative baseline in music quality.
