Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MuSViT, the first foundation vision model specifically designed for sheet music representation, utilizing a Vision Transformer (ViT) architecture pre-trained on 9.7 million pages from the IMSLP through Masked Autoencoders. The model employs a two-stage curriculum, starting with synthetic warm-up training on typeset scores followed by extensive training on the full corpus, allowing it to effectively capture the complex structure of musical notation. Evaluation across four downstream tasks demonstrates that MuSViT outperforms existing vision encoders and enhances task-specific methods, establishing it as a robust backbone for understanding sheet music.
MuSViT outperforms general-purpose vision models in sheet music tasks, revealing that they systematically fail to capture the structured properties of musical notation.
Foundation models have transformed vision and language processing by providing rich, reusable representations that transfer across diverse tasks. Sheet music, as a visual encoding of musical language, lacks such a strong domain-specific backbone. We introduce MuSViT (Music Score Vision Transformer): the first foundation vision model for sheet music representation -- a ViT encoder pre-trained via Masked Autoencoders on 9.7 million pages from the IMSLP. To handle the complexity of real-world scores, we adopt a two-stage curriculum: a synthetic warm-up on typeset scores followed by large-scale training on the full IMSLP corpus. We evaluate MuSViT on four downstream tasks -- full-page and staff-level music score recognition, music symbol detection, and score difficulty classification -- under two scenarios: linear probing (frozen encoder) and fine-tuning. Under linear probing, MuSViT consistently outperforms modern vision encoders, revealing that general-purpose representations, regardless of scale, fall systematically short on the structured symbolic properties of musical notation. Under fine-tuning, MuSViT generally improves upon task-specific state-of-the-art methods. An additional embedding-transcription consistency analysis reveals that MuSViT encodes symbolic musical structure directly in its representation space -- unlike other encoders, whose embeddings do not correlate with music notation content. These results establish MuSViT as a foundation backbone for sheet music understanding.