Search papers, labs, and topics across Lattice.
This paper introduces P-MUSE, a novel framework for MIDI-to-Music synthesis that integrates both conditional generation and In-Context Learning, allowing for flexible input options that can include or exclude aligned MIDI data. By employing a multi-stage Curriculum-Learning approach and a phase-aware classifier-free guidance scheduling principle, P-MUSE enhances control over timbre and MIDI alignment while facilitating local music editing. The authors also present a comprehensive benchmark for evaluating the framework across various tasks and instruments, demonstrating its effectiveness in generating high-quality musical outputs.
P-MUSE achieves unprecedented flexibility in music synthesis by allowing users to generate and edit music with or without aligned MIDI inputs.
MIDI-to-Music system renders the melody and rhythm of a target MIDI sequence into musical segment while cloning instrument timbre from a prompt recording. Existing systems typically adopt one of two distinct paradigms: conditional generation with prompt audio alone, which remains applicable when aligned prompt MIDI is unavailable, and In-Context Learning with paired prompt audio and MIDI, which exploits cross-modal alignment for stronger control on MIDI following and timbre similarity. We introduce P-MUSE, an instrumental MIDI-to-Music framework that unifies both paradigms via a multi-stage Curriculum-Learning supporting prompt-MIDI-optional inputs. P-MUSE further unifies music generation and local editing through a shared fill-in-the-middle formulation. Grounded in theoretical analysis and empirical study, we propose a phase-aware classifier-free guidance scheduling principle for Transcription-to-Audio systems, alongside a Tail-Drop strategy. Finally, to advance research in this field, we establish the first comprehensive benchmark, covering various prompt modes, generation/editing tasks, and four representative instruments: piano, guitar, bass, and drums. Demos are available at https://p-muse.github.io/.
