Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a masked multimodal speech synthesis framework that integrates surface electromyography (sEMG) and video-based lipreading to enhance silent speech interfaces (SSIs) for individuals with voice production impairments. By employing modality masking during training, the framework achieves significant reductions in word error rates, outperforming unimodal baselines by up to 14 absolute percentage points in multispeaker settings. The findings highlight the importance of masking strategies for robustness against modality degradation and reveal complementary contributions from both modalities, particularly in vowel and consonant recognition.
Masked multimodal integration can reduce word error rates in silent speech synthesis by up to 14 percentage points, offering a robust solution for real-world applications.
Speech restoration through silent speech interfaces (SSIs) has emerged as a promising assistive technology for individuals with impaired or absent laryngeal voice production. Among non-invasive SSI modalities, surface electromyography (sEMG) and video-based lipreading provide complementary articulatory information, yet their integration for continuous speech synthesis remains underexplored. Moreover, existing multimodal approaches rarely address robustness to modality degradation or temporary sensor failure, limiting their applicability in realistic scenarios. In this work, we propose a masked multimodal speech synthesis framework that jointly leverages sEMG and lipreading signals through modality masking during training. Under multispeaker settings, the proposed approach reduces word error rate by up to 14 absolute percentage points compared to the strongest unimodal baseline. Experimental results not only show that masking strategies are critical for these performance gains and robustness under low-bitrate conditions, but also that they generalize better than degradation-specific data augmentations in the presence of modality absence conditions. Phone-level analyses further reveal complementary contributions across modalities, with particularly strong benefits for vowels and for specific consonant groups. Overall, these findings demonstrate the effectiveness and robustness of masked multimodal integration for silent speech synthesis, although adaptation to laryngectomized speakers remains an open research challenge.