Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to speech synthesis that integrates biomechanical vocal-tract models with neural speech systems through a joint-embedding predictive architecture (JEPA). By aligning articulatory actions with a duration-preserving acoustic carrier using a reinforcement learning/cross-entropy method, the authors achieve synchronized three-dimensional vocal-tract motion that enhances the physical plausibility of generated speech. The results demonstrate significant improvements in automatic speech recognition metrics and perceptual quality, indicating a promising direction for more robust speech synthesis methods.
Achieving synchronized 3D vocal-tract motion could revolutionize how we generate and perceive synthesized speech.
Modern neural speech systems can generate intelligible waveforms, but they usually hide the physical speech-production state that produced the sound. Conversely, biomechanical vocal-tract models expose articulatory structure, contact behavior, airflow routing, and geometric constraints, but direct physical waveform synthesis remains less robust than modern neural vocoders. A duration-preserving acoustic carrier supplies the listening waveform, while a corrected three-dimensional vocal-tract model supplies synchronized jaw, lip, tongue, velum, laryngeal, oral-airflow, and nasal-airflow motion. A joint-embedding predictive architecture (JEPA)-style representation and a reinforcement learning/cross-entropy method (RL/CEM) trajectory-selection loop align articulatory actions to the acoustic carrier and to physical-plausibility constraints. The evaluation contains 12 3D recordings covering 24 minimal-pair stimuli. On the 24-word set, the carrier obtains good automatic speech recognition (ASR) results (an 8.33\% WER, a 4.17\% CER), a UTMOS score of 3.174, a mean JEPA score of 0.864, and a mean timbre-guard score of 0.947.
