Search papers, labs, and topics across Lattice.
The SKY-Piano dataset introduces a comprehensive multimodal collection of piano performance data, encompassing 11 hours of recordings that integrate audio, MIDI, motion, and video from both professional and amateur pianists. This dataset is significant as it allows for the exploration of diverse performance aspects, including hand and body motion, and provides tools for fingering annotation and MIDI-to-motion generation. Key findings include the development of a fingering annotation model and the successful demonstration of MIDI-to-motion generation, showcasing the dataset's potential for advancing music information retrieval research.
A groundbreaking multimodal dataset reveals intricate details of piano performance, enabling new avenues for music information retrieval and analysis.
Music information retrieval research on piano performance increasingly involves diverse modalities of data and annotations beyond audio and MIDI. We present SKY-Piano, a multimodal piano performance dataset that includes 11 hours of performance recordings of motion, multi-view video, audio, MIDI from 7 professional and 12 amateur pianists along with MusicXML scores. The performance pieces were selected considering playing technique, difficulty, and performer expertise on a shared core repertoire. The motion data include both hand and body motion, released in both flagged form, where samples lost to marker occlusion are marked as unreliable, and imputed form, where those gaps are reconstructed, together with Visual3D body-segment kinematics and other time-synchronized modalities. To easily browse different modalities of data at a glance, we provide an interactive web browser. In addition, we developed a fingering annotation model and tool for deriving pseudo fingering annotations from the MIDI and motion data. Lastly, we present MIDI-to-motion generation through a fine-tuning experiment as a use case of the dataset.
