Search papers, labs, and topics across Lattice.
This paper identifies a critical limitation in standard music transformers, which fail to maintain equivariance in their representations when inputs are time-shifted or pitch-transposed, particularly as model size increases. To address this, the authors introduce the Equivariant Music Transformer (EMT), which incorporates self-distillation and an auxiliary equivariance regularization loss to enhance both next-token prediction and the generation of equivariant representations. The EMT outperforms traditional methods and SOTA baselines in both objective and subjective evaluations, highlighting the necessity of dedicated inductive biases in music representation learning.
Standard music transformers lose equivariance as they scale, but the Equivariant Music Transformer recovers this crucial property, enhancing generative performance.
Humans recognize a musical passage even when it is shifted in time or transposed in pitch, indicating a notion of equivariance in the representation space. Our analysis, however, shows that standard music transformers map such time-shifted or pitch-transposed inputs onto uncorrelated representations: these models become progressively less equivariant as they scale in size or train longer. This suggests that in standard music transformers, additional model capacity is allocated to memorizing absolute patterns rather than capturing shared musical structures. In this paper, we propose the Equivariant Music Transformer (EMT), which enforces equivariance through self-distillation by jointly optimizing a next-token-prediction and an auxiliary equivariance regularization loss. We find that the additional equivariance loss acts as a beneficial regularizer, simultaneously improving next-token prediction and producing equivariant latent representations. Through both objective and subjective evaluations, EMT demonstrates superior equivariance and generative capability compared to data augmentation, feature engineering, and state-of-the-art (SOTA) baselines. More broadly, our findings reveal that standard language modeling methods alone do not capture music's translational symmetries, and dedicated inductive biases are required to produce better music representations. The code, weights and demos are available online.
