Search papers, labs, and topics across Lattice.
This paper introduces SketchSong, a hierarchical framework for song generation that improves coherence and richness by integrating explicit song-level sketch planning with fine-grained multi-track modeling. By predicting high-level sketch tokens that guide the arrangement before generating detailed audio, SketchSong effectively addresses common issues like weak transitions and limited dynamic progression. Experimental results indicate that SketchSong outperforms baseline models in both objective metrics and human evaluations, showcasing its capability without relying on additional post-training techniques.
SketchSong achieves superior song coherence and richness by explicitly planning arrangements before audio generation, outperforming strong post-trained models without extra optimization.
Recent song generation systems can synthesize realistic audio, yet generating complete songs remains challenging for two reasons. First, explicit song-level arrangement planning remains limited in existing methods, so models often need to organize overall arrangement development while generating low-level audio details. This often leads to incoherence in arrangements, such as weak section transitions and limited dynamic progression. Second, coarse modeling of different musical parts obscures their distinct roles and interactions, limiting arrangement richness of generated songs. In this paper, we present SketchSong, a hierarchical song generation framework that addresses these issues through song-level sketch planning and fine-grained multi-track modeling. Along the temporal dimension, SketchSong first predicts a compact sequence of high-level sketch tokens derived from compressed audio representations, and then generates audio tokens conditioned on these sketches. This coarse-to-fine process gives the model an explicit arrangement plan before detailed audio generation. Along the track dimension, SketchSong explicitly models four tracks, i.e., vocals, bass, drums and other instruments. This enables the model to capture the roles and interactions of different musical parts more precisely. Experiments on song generation benchmarks show that SketchSong consistently outperforms our baseline on both objective metrics and human listening tests. Despite not employing additional post-training for preference optimization such as lyrics and text-prompt alignments, SketchSong achieves competitive results against strong, post-trained open-source systems, demonstrating the effectiveness of our overall design.
