Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
The authors address audiovisual speech recognition (AVSR) data scarcity by using an audio-driven talking-head pipeline to synthesize lip-synchronized video tracks directly from audio corpora. This approach eliminates the heavy burden of recording paired multimodal datasets, creating a practical path to scale noise-robust speech models to less-resourced languages. Evaluated on Spanish and Catalan, augmenting real audiovisual data with synthetic video achieved up to a 16.2% relative WER reduction, while purely synthetic visual data successfully bootstrapped functional AVSR baselines.
Audio-driven synthetic lip movements alone can bootstrap audiovisual speech recognition baselines and slash word error rates by up to 16.2% when used as augmentation.
Audiovisual Speech Recognition (AVSR) is a multimodal approach to speech recognition that incorporates visual information from lip movements to enhance model performance. Despite its advantages, its development remains constrained by the limited availability of labeled audiovisual (AV) datasets. This work explores the use of synthetic visual data as a solution, using an audio-driven talking-head pipeline to generate lip-synchronized visual content from existing audio data. We evaluate the effectiveness of synthetic visual data both as an augmentation strategy and as a standalone training resource, applying our approach to Spanish and Catalan. Our results show that augmenting real AV data with synthetic samples yields relative Word Error Rate (WER) reductions of up to 16.2%, demonstrating the potential of this approach. Moreover, we demonstrate that synthetic data alone can serve as a baseline for AVSR training in languages lacking AV datasets. These findings provide evidence that synthetic visual data can serve as a scalable solution to AVSR data scarcity, enabling broader language coverage.