Search papers, labs, and topics across Lattice.
This paper introduces V2N, the first comprehensive Visual Piano Transcription (VPT) system that addresses the limitations of existing methods by jointly predicting onset, offset, key hold, and velocity using a shared temporal backbone. By employing multi-task supervision and per-frame training, V2N significantly enhances the accuracy of onset detection and enables precise offset and velocity predictions. The system achieves state-of-the-art performance on the PianoVAM and R3 datasets, demonstrating the effectiveness of longer temporal context in improving transcription accuracy.
Multi-task supervision in V2N allows for unprecedented accuracy in piano transcription, setting new benchmarks in onset, offset, and velocity prediction.
Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.
