Search papers, labs, and topics across Lattice.
This paper introduces a novel method for multilingual word-level forced alignment that leverages a combination of representations from the Massively Multilingual Speech (MMS) model and a self-supervised phoneme boundary detector (UnSupSeg). By employing a learned dynamic programming approach, the model effectively fuses these representations to estimate word-boundary probabilities across extensive temporal contexts. The results demonstrate significant improvements over existing alignment methods, including the Montreal Forced Aligner, achieving competitive performance on unseen languages and indicating scalability to over 1100 languages with no additional training required.
Achieving superior multilingual word alignment accuracy without the need for further training on new languages could revolutionize speech processing across diverse linguistic contexts.
We present a method for accurate multilingual word-level forced alignment, consisting of an alignment encoder and a learned alignment decoder. The encoder integrates two representations: one from the Massively Multilingual Speech (MMS) model and another from a self-supervised phoneme boundary detector (UnSupSeg). It learns to fuse them and to estimate word-boundary probabilities over long temporal contexts. The alignment decoder is a learned dynamic programming that combines encoder outputs with segmental features over the MMS and UnSupSeg representations to infer final word boundaries. Trained iteratively on TIMIT and Buckeye, the proposed approach outperforms Montreal Forced Aligner (MFA) and MMS-based alignment on both datasets. On unseen languages (Dutch, German, and Hebrew), the proposed model achieves performance consistently better than or on par with existing alignment approaches, indicating its potential to scale to 1100+ languages supported by MMS without further training.
