Search papers, labs, and topics across Lattice.
This paper introduces DLLM-VSR, a novel Visual Speech Recognition (VSR) framework based on Diffusion Large Language Models (DLLMs) that replaces autoregressive decoding with iterative masked denoising for flexible-order decoding. They use confidence-based unmasking to commit to high-confidence tokens early, providing bidirectional context for refining ambiguous tokens. By incorporating a two-stage masked-denoising training strategy and length-guided candidate decoding, DLLM-VSR achieves state-of-the-art performance on the LRS3 dataset.
Ditch the left-to-right constraint: diffusion models can now transcribe speech by iteratively refining predictions with bidirectional context, achieving state-of-the-art results.
Existing Visual Speech Recognition (VSR) systems commonly rely on left-to-right autoregressive decoding, which can force premature decisions on visually ambiguous tokens before sufficient context is available. We propose DLLM-VSR, to the best of our knowledge, the first Diffusion Large Language Model (DLLM)-based VSR framework, formulating transcription as iterative masked denoising with flexible-order decoding. With confidence-based unmasking, DLLM-VSR commits high-confidence positions early and uses the committed tokens as bidirectional context to refine ambiguous ones. To adapt DLLMs to VSR, we introduce a two-stage masked-denoising training strategy that separates visual-to-text content alignment from length modeling. We further observe a performance gap with oracle-length decoding, which assumes access to the true transcript length, indicating that reducing target-length uncertainty can improve DLLM-based VSR. To reduce this gap, we develop length-guided candidate decoding, which uses video duration to construct plausible transcript-length hypotheses, decodes under multiple hypotheses, and reranks candidates using length plausibility and decoding confidence. The proposed method achieves a state-of-the-art WER of 19.5\% on LRS3 using only its labeled training data.
