Search papers, labs, and topics across Lattice.
This paper introduces Latent Softmax, a novel output layer for phoneme-based multilingual automatic speech recognition (ASR) that effectively addresses the mismatch in supervision granularity between tonal and non-tonal languages. By treating tone-marked vowels as subclasses of base vowels while keeping consonants and CTC blanks as singleton labels, Latent Softmax enhances cross-lingual sharing of acoustic evidence. The method demonstrates significant improvements in phoneme error rates across multiple datasets, achieving reductions of up to 17.5% compared to standard softmax approaches, and also enhances word error rates in phoneme-to-grapheme conversion tasks.
Latent Softmax achieves up to 17.5% lower phoneme error rates in multilingual ASR by intelligently modeling tonal distinctions without sacrificing cross-lingual sharing.
Phoneme-based multilingual automatic speech recognition (ASR) can share acoustic evidence across languages more directly than language-specific subword modeling. When tonal and non-tonal languages are jointly trained, however, their supervision granularity does not match: tonal languages annotate tone-marked vowels, whereas non-tonal languages typically provide only base-vowel labels. A standard softmax either treats the two as unrelated classes, weakening cross-lingual sharing, or collapses tones, losing distinctions required by tonal languages. We propose Latent Softmax, a connectionist temporal classification (CTC)-compatible output layer that models tone-marked vowels as subclasses and base vowels as major classes, while consonants and the CTC blank remain singleton labels. When only a base-vowel major-class label is observed, the tone-marked vowel subclass is treated as latent and marginalized out. Multilingual experiments on AISHELL-1 Mandarin and LibriSpeech English show that Latent Softmax reduces speech-to-phoneme (S2P) phoneme error rates over a standard softmax multilingual baseline by 8.4% on AISHELL-1, 17.5% on LibriSpeech test-clean, and 12.6% on test-other. The improved speech-to-phoneme encoders also yield consistent word error rate gains for both large-language-model phoneme-to-grapheme conversion and projector-based interfaces. After code-switching adaptation, Latent Softmax further reduces projector-based mixed error rate by 2.6% on ASRU2019 and 9.5% on CS-Dialogue datasets.
Tsinghua AI