Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study systematically categorizes and quantifies the impact of Automatic Speech Recognition (ASR) substitution errors on Vietnamese speech translation, revealing that phonetic confusions are the primary source of these errors. To address this issue, the authors introduce Phonetically-Informed Data Augmentation (PiDA), which enhances training data by substituting words with phonetically similar alternatives. Fine-tuning on PiDA-augmented data results in significant improvements in translating erroneous ASR outputs, achieving up to a 2.04 BLEU score increase over standard methods while also benefiting clean-text performance.
Phonetic confusions, not random noise, are the main culprits behind ASR errors in Vietnamese speech translation, and addressing them can boost translation quality significantly.
Cascaded speech translation (ST) systems suffer from error propagation when Automatic Speech Recognition (ASR) outputs incorrect transcripts. We present the first systematic categorization of ASR errors for Vietnamese ST, classifying substitution errors by phonetic cause and quantifying their impact on downstream Neural Machine Translation (NMT) performance using Linear Mixed-Effects Modelling. We confirm that most ASR substitution errors arise from phonetic confusions rather than random noise, and that these phonetic errors significantly degrade ST quality. Motivated by this finding, we propose Phonetically-Informed Data Augmentation (PiDA), which generates ASR-like corruptions by substituting words with phonetically similar alternatives using phonetic word embeddings. Fine-tuning on a PiDA-augmented version of FLEURS Vietnamese-English improves translation of erroneous ASR outputs (up to +2.04 BLEU over standard fine-tuning) while also slightly improving clean-text performance.