Search papers, labs, and topics across Lattice.
This paper introduces an end-to-end simultaneous dysarthric speech reconstruction (DSR) system to address the latency and robustness issues of cascaded ASR-TTS systems. The system incorporates a frame-level adaptor module to improve the error tolerance of TTS to ASR outputs through explicit-implicit semantic information fusion and joint training. Additionally, a multiple wait-k autoregressive TTS module is designed to enhance prosodic feature prediction via multi-view knowledge distillation. The proposed system achieves a 54.25% relative reduction in WER and a MOS of 4.67 on the UASpeech dataset, with a real-time factor of 0.71.
Achieve near-real-time dysarthric speech reconstruction with a 54% WER reduction by jointly training ASR and TTS with a novel frame-level adaptor and knowledge distillation.
Dysarthric speech reconstruction (DSR) typically employs a cascaded system that combines automatic speech recognition (ASR) and sentence-level text-to-speech (TTS) to convert dysarthric speech into normally-prosodied speech. However, dysarthric individuals often speak more slowly, leading to excessively long response times in such systems, rendering them impractical in long-speech scenarios. Cascaded DSR systems based on streaming ASR and incremental TTS can help reduce latency. However, patients with differing dysarthria severity exhibit substantial pronunciation variability for the same text, resulting in poor robustness of ASR and limiting the intelligibility of reconstructed speech. In addition, incremental TTS suffers from poor prosodic feature prediction due to a limited receptive field. In this study, we propose an end-to-end simultaneous DSR system with two key innovations: 1) A frame-level adaptor module is introduced to bridge ASR and TTS. By employing explicitimplicit semantic information fusion and joint module training, it enhances the error tolerance of TTS to ASR outputs. 2) A multiple wait-k autoregressive TTS module is designed to mitigate prosodic degradation via multi-view knowledge distillation. Our system has an average response time of $\mathbf{1. 0 3}$ seconds on Tesla A100, with an average real-time factor (RTF) of $\mathbf{0. 7 1}$. On the UASpeech dataset, it attains a mean opinion score (MOS) of $\mathbf{4. 6 7}$ and demonstrates a $\mathbf{5 4. 2 5 \%}$ relative reduction in word error rate (WER) compared to the state-of-the-art. Our demo is available at: https://wflrz123.github.io/
