Search papers, labs, and topics across Lattice.
This paper introduces a hierarchical self-supervised world model for symbolic music, utilizing a 2.55M-parameter Swin V2 encoder trained on MIDI piano-roll images without any labels or music-theory vocabulary. The model effectively decodes musical properties at varying time scales, revealing that phrase boundaries and harmonic details can be extracted from different levels of the hierarchical representation. Notably, the inclusion of a small chord-supervision head significantly enhances chord recovery and key detection performance, demonstrating the model's potential for collaborative music creation while preserving human agency.
Chord recovery accuracy jumps from 18% to 54% with minimal supervision, showcasing a new frontier in music co-creation agents.
Collaborative music agents need internal representations rich enough to support both understanding and generation, yet flexible enough for a workflow where the human retains agency. We present a hierarchical self-supervised ``world model''for symbolic music: a 2.55M-parameter Swin V2 encoder trained on MIDI piano-roll images with JEPA-style objectives (pitch- and time-shift equivariance, masked embedding prediction, and a distributional regularizer), using no labels and no music-theory vocabulary. Probing the frozen embeddings shows that the level at which a musical property becomes decodable tracks its musical time scale: phrase boundaries are read off the coarsest levels, note density and harmonic detail off the finest. Temporal and phrase structure emerge from the self-supervised objectives alone, while harmonic content must be asked for; a small chord-supervision head raises joint chord recovery from .18 to .54, and key detection, which is never supervised, from .16 to .70. Following the Representation AutoEncoder paradigm, a conditional flow-matching model stands in for a trained decoder, flowing in pixel space from PCA-reduced conditioning: it reproduces a target window at pixel F1 $0.996$, and the same per-level conditioning dropout that controls how far variations stray also enables graphical prompting for masked inpainting with no inpainting-specific sampler. The pipeline runs on CPU producing a suggestion in $2.8$ s, or $0.6$ s on Apple MPS, which we demonstrate in a live interactive demo. In concert with an LLM-based brain, these capabilities supply the core of a collaborative music creation agent in service of, rather than in place of, human agency.
