Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for real-time interactive music generation that transforms static generative models into dynamic instruments capable of responding to human expression. By employing a streaming autoregressive latent space and utilizing prompt-only inputs, the approach eliminates the need for costly paired audio-latent datasets while ensuring high acoustic fidelity through music-aware consistency objectives. The result is a system that allows musicians to interactively steer musical trajectories in real-time, fundamentally changing the landscape of live performance and composition with AI.
Real-time interactive music generation is now possible, allowing musicians to steer AI compositions on the fly without latency or loss of acoustic fidelity.
Interactive music and live performance relies on real-time human expression, but modern generative music AI remains largely absent from this domain due to its prohibitive inference latency and offline rendering paradigm. To provide pioneer musicians with a novel medium for interactive composition, we should fundamentally change these static models into dynamic, playable instruments. In this paper, we propose a framework that bridges this gap. To achieve the low latency required for live interaction without sacrificing structural coherence, we formulate distillation within a streaming autoregressive latent space. Our approach gets rid of the need for expensive paired audio-latent datasets by utilizing prompt-only inputs to synthesize teacher-guided, chunk-wise trajectories on the fly. Because live instruments require high acoustic fidelity, we introduce music-aware consistency objectives, which combine latent, spectral, and temporal-difference losses, to preserve crucial qualities like timbre, transients, and rhythmic stability during accelerated single-step streaming generation. Implemented via parameter-efficient adaptation, our distillation reduces generation steps to achieve a low real-time factor. Crucially, by operating as a continuous autoregressive stream, the system can seamlessly assimilate dynamic human inputs on the fly, allowing users to instantly steer the musical trajectory without interrupting the audio flow. Ultimately, this work recontextualizes generative text-to-music models not as passive prompt-and-wait systems, but as responsive instruments, opening new frontiers for live human-AI musical co-creation.