Search papers, labs, and topics across Lattice.
This paper introduces FullDiT, a novel conditional diffusion model for music generation that addresses codec-interface exposure bias by treating rendering as full-context generation rather than mere reconstruction. By integrating eight frame-aligned RVQ streams with independently encoded lyrics and captions, and employing Error-Matched Distractor Conditioning (EMDC) during training, FullDiT significantly enhances the fidelity of generated music. The system outperforms five commercial music generation systems across multiple metrics, showcasing its effectiveness in producing high-quality audio outputs.
FullDiT not only outperforms leading commercial music generators but also redefines how we approach music rendering by leveraging full-context generation techniques.
Hybrid music generators combine the long-range planning of an autoregressive language model with the fidelity of a diffusion- or flow-based acoustic renderer. Yet renderers are trained with clean, target-derived codec tokens but deployed with imperfect language-model predictions, creating codecinterface exposure bias. Rather than treating rendering as a simple reconstruction task,we formulate it as full-context generation from an imperfect discrete plan.We introduce FullDiT, a conditional DiT that fuses eight frame-aligned RVQ streams with independently encoded captions and lyrics and uses non-causal self-attention over the complete acoustic latent sequence. During training, Error-Matched Distractor Conditioning (EMDC) matches per-codebook replacement rates to teacher-forced top-1 error rates and samples near-miss tokens from cosine-KNN neighborhoods without changing the acoustic target. At inference, four-way classifier-free guidance (4-CFG) independently scales codec, lyric, and caption guidance increments. Matched ablations show that EMDC improves ViSQOL by 0.77 under synthetic corruption and is clearly preferred in non-tied comparisons with fixed languagemodel tokens. Further ablations show gains from full-song context and renderer-side text conditioning. The complete system outperforms five commercial systems on 15 of 18 automatic metrics and ranks among the top three on the Artificial Analysis Music with Vocals Leaderboard. The demo page is available at https://selinacloudl.github.io/fulldit-demo/.
DAMO