Search papers, labs, and topics across Lattice.
The paper introduces Depth-Structured Music Recurrence (DSMR), a recurrent Transformer architecture for long-context symbolic music modeling that uses segment-level recurrence with detached cross-segment states to extend context beyond fixed-length excerpts. DSMR employs a layer-wise memory-horizon schedule to budget recurrent KV states across depth, creating depth-dependent temporal receptive fields without increasing compute depth. Experiments on the MAESTRO dataset demonstrate that a two-scale DSMR schedule, allocating longer history windows to lower layers and a uniform short window to upper layers, achieves a practical quality-efficiency trade-off for full-length music modeling under limited resources.
Achieve state-of-the-art symbolic music modeling on long sequences with a novel recurrent attention mechanism that cleverly budgets memory across transformer layers, enabling efficient processing on resource-constrained devices.
Long-context modeling is essential for symbolic music generation, since motif repetition and developmental variation can span thousands of musical events. However, practical composition and performance workflows frequently rely on resource-limited devices (e.g., electronic instruments and portable computers), making heavy memory and attention computation difficult to deploy. We introduce Depth-Structured Music Recurrence (DSMR), a recurrent long-context Transformer for full-piece symbolic music modeling that extends context beyond fixed-length excerpts via segment-level recurrence with detached cross-segment states, featuring a layer-wise memory-horizon schedule that budgets recurrent KV states across depth. DSMR is trained in a single left-to-right pass over each complete composition, akin to how a musician experiences it from beginning to end, while carrying recurrent cross-segment states forward. Within this recurrent framework, we systematically study how depth-wise horizon allocations affect optimization, best-checkpoint perplexity, and efficiency. By allocating different history-window lengths across layers while keeping the total recurrent-state budget fixed, DSMR creates depth-dependent temporal receptive fields within a recurrent attention stack without reducing compute depth. Our main instantiation is a two-scale DSMR schedule that allocates long history windows to lower layers and a uniform short window to the remaining layers. Experiments on the piano performance dataset MAESTRO demonstrate that two-scale DSMR provides a practical quality--efficiency recipe for full-length long-context symbolic music modeling with recurrent attention under limited computational resources.
