Search papers, labs, and topics across Lattice.
This paper introduces a speaker-disentangled syllabic tokenizer that addresses the challenge of unsupervised syllabic tokenization by regressing speaker-perturbed student representations towards clean teacher targets within fixed-length chunks. The method overcomes the limitations of previous approaches that led to models predicting speaker identity instead of linguistic content, thereby enhancing the purity of syllabic tokens. Experimental results show that this approach not only achieves state-of-the-art performance in syllable boundary detection and syllabic segment clustering but also improves a speech language model's syntactic and semantic understanding by 7% relative to existing phone-level models.
Speaker-disentangled tokenization boosts syllabic segment accuracy and enhances language model performance, revealing a critical link between speaker identity and linguistic content.
Unsupervised syllabic tokenization aims to learn discrete syllabic tokens that capture latent linguistic content-related structure from raw speech. Recent syllabic tokenization methods employ teacher-student distillation of the pretrained HuBERT to organize latent speech frame representations into syllabic segments. However, when trained with an utterance-level cross-entropy objective, the model predicts speaker identity rather than linguistic content, thereby compromising the purity of syllabic tokens. To address this problem, we propose a speaker-disentangled syllabic tokenizer that regresses speaker-perturbed student representations toward clean teacher targets within fixed-length chunks. Experimental results demonstrate that our proposed method achieves state-of-the-art performance in syllable boundary detection and syllabic segment clustering. Moreover, a speech language model trained on our syllabic tokens achieves a 7% relative improvement in syntactic and semantic understanding over the phone-level SpiRit-LM.
