Search papers, labs, and topics across Lattice.
The paper introduces EventSpeech, a text-conditioned speech generation framework that leverages neuromorphic event cameras to capture high-frequency articulatory transients, overcoming the limitations of RGB cameras. They propose an architecture with an Event Encoder and a Hierarchical Wavelet Contextualizer within the Audio Encoder, using a bidirectional alignment mechanism. Experiments on the newly constructed EVT-SPK dataset demonstrate that EventSpeech outperforms existing methods in preserving fine-grained emotions and resisting motion blur.
Neuromorphic event cameras can unlock a new level of expressiveness in speech generation, capturing subtle emotional nuances missed by traditional RGB-based methods.
Traditional RGB-based speech generation faces Temporal Granularity Mismatch since fixed camera exposure times inevitably blur the high-frequency articulatory transients essential for rendering emotional speech. To break this ceiling, we propose EventSpeech as a novel text-conditioned framework pioneering the use of neuromorphic events for expressive speech generation, since these microsecond-precise events naturally align with acoustic waveform dynamics. Our architecture integrates a dedicated Event Encoder to model sparse neuromorphic events alongside a multi-scale Audio Encoder featuring a Hierarchical Wavelet Contextualizer (HWC). A bidirectional alignment mechanism seamlessly synchronizes linguistic content and visual dynamics with dense acoustic features. Furthermore, we construct EVT-SPK as the first benchmark comprising large-scale synthetic data and real-world recordings from specialized neuromorphic hardware. Extensive evaluations demonstrate that EventSpeech significantly outperforms current baselines by preserving fine-grained emotions and resisting motion blur to establish a new paradigm for multimodal speech generation. Code and demo are available at https://xrfang-0102.github.io/EventSpeechWeb/.
