Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MPEcho, a novel framework for controllable cover song generation that enhances the preservation of both melodic and linguistic elements from reference songs. By integrating a phoneme encoder and a length regulator into the existing SongEcho model, MPEcho provides explicit phoneme-level conditioning, which significantly reduces the phoneme error rate (PER) associated with lyric accuracy. Experimental results demonstrate that MPEcho, supported by the Phonsa transcription model for high-precision phoneme annotations, outperforms previous methods in generating accurate cover songs.
MPEcho slashes phoneme error rates in cover song generation by integrating explicit phoneme conditioning, transforming how we think about lyrical accuracy in music synthesis.
Cover song generation (CSG) should preserve the melodic and linguistic content of a reference song while recreating the remaining musical components. The state-of-the-art model SongEcho utilizes $F_0$ sequences and voiced/unvoiced (V/UV) tags for conditioning; however, implicit linguistic information from V/UV tags cannot guarantee lyric accuracy, leading to a high phoneme error rate (PER). Inspired by singing voice synthesis (SVS), we propose MPEcho, which integrates a phoneme encoder and a length regulator (LR) into the SongEcho framework. By providing explicit phoneme-level conditioning and precise temporal boundaries, MPEcho significantly reduces PER. To enable this, we developed Phonsa, a Whisper-based automatic transcription model that provides high-precision phoneme-level annotations for singing voices, overcoming the scarcity of high-quality audio-phoneme pairs. Experimental results validate the effectiveness of Phonsa for alignment and MPEcho for end-to-end CSG. The audio samples, code and weights can be accessed from https://lonian6.github.io/MPEcho.github.io/.