Search papers, labs, and topics across Lattice.
This paper introduces Co-policy, a framework that enhances human-robot musical co-creation by effectively linking semantic musical understanding with real-time physical performance. By utilizing pre-inference semantic anchors and a fine-tuned Qwen-vl planner, Co-policy transforms various inputs into structured co-creation plans while employing a Gaussian-Mixture Visuomotor Policy for low-latency execution. Experimental results demonstrate that Co-policy significantly outperforms existing robotic playback systems in intent alignment, execution accuracy, and response frequency, highlighting its potential for embodied human-AI collaboration in music.
Co-policy enables robots to generate musically complementary responses in real-time, transforming the landscape of human-robot musical collaboration.
Art has long stood as a pivotal expression of human creativity. Embodied artificial intelligence offers a route for generative models to participate in that creativity through physical action rather than disembodied digital content. In robotic music co-creation, it is challenging to connect semantic musical understanding with real-time and physically executable performance. We present Co-policy, a framework for human-robot musical co-creation that separates semantic intent grounding, constrained musical variation, and visuomotor execution. To ground musical semantics, Co-policy uses pre-inference semantic anchors and a fine-tuned Qwen-vl planner (F-Qwen) to transform speech, live musical seeds, and visual observations into structured co-creation plans. To support low-latency execution, Co-policy introduces a Gaussian-Mixture Visuomotor Policy (GMP), implemented as a conditional mixture-density policy that maps target notes and visual context to multimodal robot actions in a single forward pass. Unlike robotic playback systems that merely reproduce user-specified notes, Co-policy generates complementary musical responses under both musical and physical constraints. Real-robot chime experiments, ablations, and expert evaluation show improved intent alignment, execution accuracy, and response frequency over diffusion-policy and ablated baselines, supporting physically grounded action generation as a key requirement for embodied human-AI co-creation.
