Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for event-based lip reading that enhances temporal modeling through two key innovations: Trajectory-Aware Differential Aggregation (TDA) for local temporal modeling and Viseme-Guided Aggregation (VGA) for improved word recognition. By optimizing temporal representations with viseme-aware supervision and employing an EMA teacher-student training strategy, the method addresses the limitations of existing approaches that compress spatial information too early. Experiments on the DVS-Lip benchmark show significant improvements in recognition accuracy, underscoring the importance of preserving temporal dynamics in visual speech recognition.
Event-based lip reading accuracy improves dramatically when leveraging viseme-aware temporal modeling, revealing the critical role of motion trajectories in distinguishing lip movements.
Event-based lip reading has recently emerged as a promising direction for visual speech recognition, benefiting from the high temporal resolution and motion sensitivity of event cameras. However, existing methods typically perform spatial compression before sufficient temporal modeling, which may suppress sparse and localized motion trajectories that are crucial for distinguishing similar lip movements. Moreover, most current approaches optimize temporal representations mainly at the word-classification level, leaving the underlying articulatory structure weakly constrained. To address these limitations, we propose a temporally enhanced framework for event-based lip reading. First, we introduce Trajectory-Aware Differential Aggregation (TDA), which performs local temporal modeling at each spatial location before adaptive spatial aggregation. Second, we propose Viseme-Guided Aggregation (VGA), a unified temporal module composed of a CTC decoder and a viseme-guided gated aggregation branch, which injects viseme-aware sequence supervision and improves final temporal aggregation for word recognition. Third, we incorporate an EMA teacher--student training strategy to enhance robustness under strong event perturbations. Experiments on the DVS-Lip benchmark verify the effectiveness of the proposed design, and extensive ablation studies further validate the contributions of TDA, VGA, and teacher--student consistency. Qualitative decoding results also demonstrate that the proposed CTC-based temporal modeling learns meaningful viseme-aware structure from event streams.
