Search papers, labs, and topics across Lattice.
This paper introduces StreamMUSE, a novel inference system designed for frame-synchronous streaming inference in language models, specifically targeting applications that require real-time synchronization with external signals, such as live music accompaniment. By employing a client-server architecture, StreamMUSE enables continuous high-frequency inference requests that align model outputs with an external clock, effectively addressing the challenges posed by varying round-trip latencies. Experimental results indicate a strong correlation between the system's real-time performance and the quality of the generated music, validating the framework's efficacy in diverse deployment environments.
Real-time synchronization of language model outputs with external signals can significantly enhance applications like live music accompaniment, achieving both speed and quality.
Language models (LMs) have become one of the most prominent paradigms in modern generative modeling. While making them faster has been the main focus of real-time deployment, speed alone is not enough. Many real-world applications, such as synchronized translation and voice synthesis, also require precise alignment between generation and external signals, both in terms of generation content and timing. We refer to this problem as \textit{frame-synchronous streaming inference}. To address it, we present StreamMUSE, an inference system that performs LM generation in response to an external signal stream within a client-server architecture. The client continuously sends high-frequency inference requests based on the most recent inputs and receives outputs synchronized to the external clock, while the server executes model inference. We demonstrate the framework through a live music accompaniment task, showing how real-time synchronization can be achieved across different deployment environments with varying round-trip latencies. We further model the relationship between system hyperparameters and round-trip latency, and evaluate how different environments affect optimal configurations to achieve real-time performance. Experimental results show a consistent correspondence between system real-time performance and music quality, demonstrating the effectiveness of the proposed framework. The project is open source. Relevant code and the latest updates are available at https://stream-muse-webpage.vercel.app/#audio-library.
