Search papers, labs, and topics across Lattice.
This paper introduces LipsFlow, a neuromorphic-inspired framework for Visual Speech Recognition (VSR) that transforms RGB videos into high-temporal-resolution event streams to address challenges posed by rapid head movements and occlusions in multi-speaker environments. By leveraging ByteTrack for tracking and TalkNet for active speaker detection, LipsFlow segments scenes into single-speaker clips, allowing for precise analysis of articulatory dynamics. The framework achieves a state-of-the-art word error rate (WER) of 22.3% at just 240 ms latency, showcasing its robustness and efficiency in handling complex visual degradation scenarios.
Achieving a 22.3% word error rate with just 240 ms latency, LipsFlow redefines the capabilities of Visual Speech Recognition in challenging multi-speaker environments.
Visual Speech Recognition (VSR) tasks in complex multi-speaker scenarios are severely hindered by rapid head motions, occlusions, and subtle lip articulations. Traditional RGB-based methods struggle here due to low rates and motion blur of frames. To overcome these, we propose LipsFlow, a neuromorphic-inspired VSR framework that converts RGB videos into high-temporal-resolution event streams. For multi-speaker, we employ ByteTrack tracking and TalkNet active speaker detection to temporally segment scenes into single-speaker clips, enabling focused per-speaker analysis. By explicitly capturing microsecond-level articulatory dynamics via learnable event-based representations, LipsFlow achieves inherent robustness against visual degradation. To efficiently model these dense event-based features and adapt to speaker-specific articulatory patterns, we introduce Optimal Transport Conditional Flow Matching (OT-CFM). It enforces deterministic, straight-line trajectory generation in a semantic latent space, slashing inference latency to just two Ordinary Differential Equation (ODE) steps. Furthermore, we design a Dual-Level Semantic Supervision mechanism combining token-level BERT weight tying and sentence-level priors to resolve homophene ambiguities. Validated on competitive benchmarks, LipsFlow achieves a state-of-the-art WER of 22.3\% at 240 ms latency, establishing a highly robust and efficient paradigm for event-based VSR.
Tsinghua AI