Search papers, labs, and topics across Lattice.
This study investigates how self-supervised learning (SSL) speech models encode articulatory information by analyzing EMA data from bilingual Finnish-Russian speakers. The findings reveal that multilingual models significantly outperform monolingual counterparts, achieving strong correlations (Pearson r up to 0.68) between SSL representations and articulatory movements, particularly in intermediate layers. Notably, tongue movements are more accurately predicted than lip movements, and task structure influences model performance, highlighting the models' potential for practical applications in speech technology.
Multilingual SSL speech models can predict articulatory movements with surprising accuracy, outperforming monolingual models even with minimal training data.
SSL speech models capture rich phonetic, prosodic, and acoustic patterns from raw audio, yet how they encode articulatory information across diverse languages remains unclear. Using EMA data from bilingual Finnish-Russian speakers, we evaluate cross-lingual correlations between SSL latent representations and articulatory movements. Models achieve strong prediction performance (Pearson r up to 0.68) even with approximately 5 minutes of training data, with multilingual models outperforming monolingual ones. Intermediate layers encode articulatory features most effectively, and tongue movements are more predictable than lip movements. We also assess the impact of task type (read versus spontaneous speech) and language proficiency, finding higher accuracy for structured tasks and strong generalization across proficiency levels. These results enhance the interpretability of SSL models and show their potential for speech-technology applications.
