Search papers, labs, and topics across Lattice.
This paper introduces TUTTI, a novel approach to audio-to-score (A2S) transcription that utilizes a fully synthetic dataset generated by a symbolic music model to overcome the limitations of scarce real-world paired data. By training a Transformer encoder-decoder on this large-scale multi-instrumentation corpus, TUTTI achieves superior foundational representations compared to traditional single-instrumentation methods. The results demonstrate that TUTTI not only sets new state-of-the-art performance across various A2S benchmarks but also exhibits impressive cross-instrument transferability, adapting effectively to previously unseen instruments.
TUTTI achieves unprecedented A2S performance by leveraging a synthetic dataset, outperforming traditional methods and enabling robust cross-instrument adaptability.
Generalizable Audio-to-Score (A2S) transcription is fundamentally constrained by the severe scarcity of high-quality, real-world paired data. Relying solely on existing human-annotated datasets often restricts the generalization of A2S models, limiting their efficacy primarily to single-instrumentation domains. To break this dependency on scarce real-world data, we introduce TUTTI (Transformer for Unified audio-To-score Transcription trained on Synthetic multi-Instrumentation Data), a pre-training paradigm driven by a purely synthetic, large-scale dataset. Rather than using human-composed scores, we leverage a symbolic music generation model to generate a massive, highly scalable multi-instrumentation corpus and create audio-score pairs with expressive acoustic characteristics. Capitalizing on the generated data, we employ a standard Transformer encoder-decoder architecture. We empirically demonstrate that pre-training a unified attention-based model on generated, multi-instrumentation data yields a consistently stronger foundational representation than single-instrumentation training. When fine-tuned with downstream real-world datasets, TUTTI outperforms previous approaches, establishing new overall state-of-the-art results across various A2S baselines. Notably, TUTTI shows remarkable cross-instrument transferability, effectively adapting to unseen instruments with highly competitive performance. The source code and the TuttiCorpus dataset will be made publicly available at https://github.com/a-musiclover/TUTTI.
Tsinghua AI