Search papers, labs, and topics across Lattice.
The paper introduces VSRo-200, a pioneering Romanian visual speech recognition dataset featuring 200 hours of podcast videos, annotated with both pseudo-labels from a fine-tuned ASR model and human transcriptions for comparative analysis. The study reveals that while human annotations yield superior performance at fixed data scales, pseudo-labels facilitate scalability and continuous improvement. Additionally, the dataset enables robust evaluations of audio-visual speech recognition under domain shifts and noisy conditions, demonstrating significant advantages of multimodal fusion over audio-only approaches.
Human annotations boost performance, but pseudo-labels unlock scalability in visual speech recognition鈥攚hat鈥檚 the trade-off?
We introduce VSRo-200, the first large-scale dataset for visual speech recognition (lip reading) in Romanian, comprising 200 hours of real-world podcast videos. All samples are annotated with pseudo-labels generated by a fine-tuned Romanian ASR model, while a subset of 100 hours is additionally transcribed by humans, enabling controlled analysis of supervision quality under a unified framework. Building on this dataset, we establish a benchmark for visual speech recognition in low-resource settings. We systematically study the impact of supervision quality, showing that while human annotations provide better performance at fixed data scales, pseudo-labels enable continued improvements through scalability. We further evaluate robustness under domain shift using curated out-of-distribution (OOD) test sets, and analyze audio-visual speech recognition (AVSR) under noisy conditions, where multimodal fusion significantly improves robustness compared to audio-only models. Finally, we demonstrate that representations learned on VSRo-200 transfer effectively to the LRRo benchmark for isolated word recognition, substantially outperforming previously reported results. Overall, VSRo-200 provides a new testbed for studying supervision, domain generalization, and multimodal fusion in low-resource visual speech recognition.
