Search papers, labs, and topics across Lattice.
The paper addresses the limitations of Unified Speech Recognition (USR), specifically its slow autoregressive pseudo-labeling and susceptibility to errors under distribution shifts. They introduce CTC-driven teacher forcing, using greedily decoded CTC outputs as input to the decoder for attention target generation in a single forward pass, enabling faster and more robust knowledge transfer. The resulting USR 2.0, incorporating mixed sampling, achieves state-of-the-art results on LRS3, LRS2, and WildVSR while halving training time and improving out-of-distribution robustness.
Ditch slow autoregressive pseudo-labeling: CTC-driven teacher forcing slashes USR training time in half while boosting robustness and accuracy in speech recognition.
Unified Speech Recognition (USR) has emerged as a semi-supervised framework for training a single model for audio, visual, and audiovisual speech recognition, achieving state-of-the-art results on in-distribution benchmarks. However, its reliance on autoregressive pseudo-labelling makes training expensive, while its decoupled supervision of CTC and attention branches increases susceptibility to self-reinforcing errors, particularly under distribution shifts involving longer sequences, noise, or unseen domains. We propose CTC-driven teacher forcing, where greedily decoded CTC pseudo-labels are fed into the decoder to generate attention targets in a single forward pass. Although these can be globally incoherent, in the pseudo-labelling setting they enable efficient and effective knowledge transfer. Because CTC and CTC-driven attention pseudo-labels have the same length, the decoder can predict both simultaneously, benefiting from the robustness of CTC and the expressiveness of attention without costly beam search. We further propose mixed sampling to mitigate the exposure bias of the decoder relying solely on CTC inputs. The resulting method, USR 2.0, halves training time, improves robustness to out-of-distribution inputs, and achieves state-of-the-art results on LRS3, LRS2, and WildVSR, surpassing USR and modality-specific self-supervised baselines.
