Search papers, labs, and topics across Lattice.
This study investigates the impact of joint language-district supervision on self-supervised speech encoders, specifically Whisper-base and Wav2Vec2.0-base, by comparing it to language-only supervision in classification tasks. The results reveal that incorporating geographical context significantly enhances district discrimination while maintaining robust language classification. Analysis using Normalized Conditional Mutual Information (NCMI) indicates that this approach fosters distinct global language clusters with subclusters that reflect geographical variation, thereby improving the embeddings' geographical separability without compromising language organization.
Joint language-district supervision not only boosts district discrimination but also preserves language classification integrity, revealing a nuanced structure in speech embeddings.
Self-supervised speech encoders are often fine-tuned with language supervision, which can overlook geographical variation. To understand the learned representations under joint supervision of language and district compared to language-only supervision, we fine-tune Whisper-base and Wav2Vec2.0-base for classification tasks with joint language-district (386 classes) and language-only classification (60 languages). The language-district supervision improves district discrimination conditioned on language in the embedding space while strong marginal language classification. We analyze the structure of the learned embeddings using Normalized Conditional Mutual Information (NCMI), showing that language-district supervision produces global language clusters with structured within language subclusters aligned to district variation, enhancing geographical separability without degrading language-level organization.
