Search papers, labs, and topics across Lattice.
This paper introduces a novel representation learning framework that enhances electrolaryngeal (EL) speech conversion to normal speech by integrating speech and text representations within a sequence-to-sequence voice conversion model. By employing a two-stage methodology that includes representation integration and reconstruction training, the authors address the cumulative mapping errors that hinder EL speech intelligibility. Experimental results show that their approach, which incorporates auxiliary text information and various fusion strategies, significantly outperforms traditional methods that rely solely on speech representations, leading to improved mapping and reconstruction quality.
Integrating text and speech representations can dramatically enhance the intelligibility of electrolaryngeal speech, outperforming traditional methods by leveraging auxiliary information.
Objective: laryngectomees depend on an electromechanical device to generate electrolaryngeal (EL) speech. Compared with normal speech, EL speech suffers from severe distortion, limited phonetic variation, unnatural prosody, and temporal shifts, degrading naturalness and intelligibility. Although sequence-to-sequence (seq2seq) voice conversion (VC) based EL-speech-to-normal-speech conversion (EL2SP) is promising, substantial mismatches between EL and normal speech inevitably cause cumulative mapping errors that limit performance. To address this, we describe a novel representation learning framework integrating speech and text representations to improve mapping and reconstruction quality within a seq2seq VC model. Methods: our methodology comprises two main stages: 1) representation integration and learning, and 2) reconstruction training. A network capable of incorporating auxiliary text information is first constructed with pretrained modules to learn speech--text-based integrated representations. Then, an autoencoder-style reconstruction strategy finalizes EL2SP model to inherit these representations without increasing model complexity. We introduce three fusion strategies including middle-, input-, and hybrid-level fusion strategies that progressively enhance learning. Moreover, besides standard seq2seq VC objectives, an additional reconstruction loss on the integrated representation is introduced to refine representation transfer. Results: experiments under different EL2SP datasets consistently demonstrate that our methods, combined with data augmentations, outperform baselines relying solely on speech representations. Furthermore, progressive improvements with system design depth validate the effectiveness of our methods. Significance: the proposed methods provide an extensible and practical methodology for EL speech enhancement and assistive communication technologies.
