Search papers, labs, and topics across Lattice.
This paper introduces Calliphony, a novel interface that integrates calligraphy with real-time generative music performance by capturing brush motion through an attachable sensor to produce multi-track MIDI. The system employs a low-latency pipeline that translates calligraphic movements into control signals, influencing musical elements such as note density and harmony in live settings. Key findings demonstrate that this approach not only enhances the expressiveness of generative music but also expands the artistic scope of calligraphy into an interactive audiovisual domain.
Calliphony transforms brush strokes into dynamic control signals for generative music, merging visual art with live performance in unprecedented ways.
While music generative models have recently gained significant attention, how they can be effectively integrated into live music performances still requires further exploration. This paper presents Calliphony, a calligraphy-driven interface for real-time generative music performance. Specifically, we build a low-latency pipeline that captures brush motion with an attachable sensor and maps it to control signals for real-time symbolic music generation. Using a generative model, the system produces multi-track MIDI in performance settings, while brush-derived control signals constrain event timing and activate additional musical layers. The generated melody is then extended with real-time harmony and additional voices, and finally rendered through a DAW for live staging. Calliphony contributes: (1) a performance-oriented prototype that uses calligraphic motion as an external control layer for a real-time symbolic music generation model, controlling note density, pitch constraints, and accompaniment-layer activation; and (2) a cross-modal performance scenario that extends calligraphy beyond a primarily visual practice into an audiovisual, AI-assisted setting.
