Search papers, labs, and topics across Lattice.
This paper introduces a unified framework for generating full-length songs from various inputs, including lyrics and musical attributes, utilizing a combination of hierarchical autoregressive modeling and flow-matching techniques. The system comprises a semantic-aware tokenizer, a hybrid language model (hybird-LM), FullDiT for flow-based audio rendering, and a two-level melody module for cover song generation, which preserves melodic integrity while allowing stylistic reinterpretation. Experimental results demonstrate that this approach achieves competitive performance across multiple song generation tasks, highlighting its effectiveness in producing high-quality music outputs.
Achieving high-quality full-length song generation from diverse inputs, this framework outperforms existing methods in musicality and fidelity.
In this report, we present a unified song generation framework capable of producing high-quality full-length music from lyrics, text descriptions, and musical attributes. The proposed framework supports three tasks: Lyrics-to-Song Generation, which generates complete songs from text descriptions, lyrics, and musical attributes; Instrumental Music Generation, which creates music without vocals; and Cover Song Generation, which reinterprets existing songs with different styles while preserving their melodic content. Architecturally, our system consists of four main components: a semantic-aware tokenizer, hybird-LM, FullDiT, and a two-level melody module. The tokenizer encodes audio into 8-codebook RVQ tokens for efficient discrete music representation. Based on these tokens, hybird-LM performs hierarchical autoregressive audio-token modeling for full-song generation. To improve audio fidelity, FullDiT performs full-song flow matching in a continuous VAE latent space conditioned on codec tokens, lyrics, and text captions. For cover song generation, the melody module extracts and discretizes melody cues from reference audio to guide generation while preserving the original melodic content. Finally, we investigate DPO, GRPO, and OPD as reward-based post-training strategies for hybird-LM and apply flow-based GRPO to FullDiT to improve musicality and rendering quality. Experimental results on a multilingual automatic benchmark, complemented by the Artificial Analysis Music with Vocals leaderboard, show that the proposed framework achieves competitive performance in the evaluated settings.
