Search papers, labs, and topics across Lattice.
The paper addresses the challenge of building parallel corpora for low-resource machine translation by focusing on source-side data curation. They introduce LALITA, a framework that leverages lexical and linguistic features to select informative source sentences for creating efficient parallel text. Experiments on English-Hindi translation, simulating low-resource scenarios, demonstrate that training on curated datasets, especially those containing complex sentences, significantly improves translation quality and reduces data requirements by more than half across multiple languages.
Cut your MT training data in half: a new framework uses linguistic features to select only the most informative source sentences for low-resource translation.
Data curation is a critical yet under-researched step in the machine translation training paradigm. To train translation systems, data acquisition relies primarily on human translations and digital parallel sources or, to a limited degree, synthetic generation. But, for low-resource languages, human translation to generate sufficient data is prohibitively expensive. Therefore, it is crucial to develop a framework that screens source sentences to form efficient parallel text, ensuring optimal MT system performance in low-resource environments. We approach this by evaluating English-Hindi bi-text to determine effective sentence selection strategies for optimal MT system training. Our extensively tested framework, (Lexical And Linguistically Informed Text Analysis) LALITA, targets source sentence selection using lexical and linguistic features to curate parallel corpora. We find that by training mostly on complex sentences from both existing and synthetic datasets, our method significantly improves translation quality. We test this by simulating low-resource data availabilty with curated datasets of 50K to 800K English sentences and report improved performances on all data sizes. LALITA demonstrates remarkable efficiency, reducing data needs by more than half across multiple languages (Hindi, Odia, Nepali, Norwegian Nynorsk, and German). This approach not only reduces MT systems training cost by reducing training data requirement, but also showcases LALITA's utility in data augmentation.
