Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a real-time musical interface that transforms natural language descriptions into dynamic procedural soundscapes, allowing performers to manipulate audio parameters directly without re-prompting. The system employs a categorical schema to ensure that generated sound configurations remain musically coherent, utilizing three interchangeable backends for flexibility in performance. Evaluation shows that the retrieval-based configuration method significantly enhances text-audio semantic alignment compared to random configurations, paving the way for more interactive and responsive musical experiences.
Real-time control over procedural soundscapes allows performers to shape music interactively, transforming text prompts into evolving audio without interruption.
We present a real-time musical interface that converts natural-language scene descriptions into evolving procedural soundscapes. A performer types a prompt such as"warm jazz cafe at midnight"and steers it through direct parameter adjustments - stepping brightness down, switching a rhythm style - each producing a predictable, audible shift without re-prompting. Where GPU-bound text-to-audio systems synthesize monolithic waveforms, our instrument generates human-readable configurations over a categorical schema, enabling fine-grained performer control; most valid combinations are designed to sound musically coherent. Three interchangeable backends - embedding retrieval for sub-second CPU-only use, hosted LLMs via API, and a fine-tuned 270M local model - all emit the same schema. A live generator architecture continuously emits audio while resolving new instructions in the background, crossfading seamlessly when ready; even when an LLM takes 5-12 seconds to respond, the audience hears uninterrupted sound - reframing text-to-music as an ongoing performable stream rather than a one-shot generation. We evaluate text-audio semantic alignment using LAION-CLAP on held-out prompts as a technical proxy, finding that retrieval-based configuration outperforms random valid configurations on this metric, while noting that LAION-CLAP also informed retrieval-map construction. We report performance observations, informal listener feedback, and release materials for the SDK, dataset artifacts, model, and audiovisual performance interface.