Search papers, labs, and topics across Lattice.
This study addresses the limitations of existing automatic music transcription methods that struggle with multi-instrument recordings by analyzing the effectiveness of synthetic data in pre-training models. By combining this approach with fine-tuning on real audio and employing reinforcement learning for post-training, the authors enhance the model's ability to generalize to complex music mixes. The resulting MuScriptor model, which incorporates conditioning on instrument presence, demonstrates improved transcription accuracy across diverse musical genres and is made available as an open-weight resource.
MuScriptor achieves unprecedented accuracy in transcribing multi-instrument music, outperforming existing models that rely solely on synthetic data.
Existing methods for automatic music transcription are often limited to single-instrument recordings or fail on complex, real music mixes. Although previous work utilizes synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings. In this work, we analyze the effectiveness of synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement learning. We further introduce conditioning on instrument presence to customize transcriptions. Finally, we release MuScriptor, an open-weight multi-instrument music transcription model that works on real-world music recordings from across a diverse range of musical genres.
