Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces BhashaSetu, a new English-Marathi parallel dataset of 2.78 million sentence pairs designed to address data scarcity in low-resource machine translation. They benchmarked several SOTA translation models and found that corpus-level deduplication during preprocessing significantly boosts translation quality for the morphologically rich Marathi language. The dataset and findings are released to encourage further research in linguistically informed low-resource NMT.
Deduplicating your training data is the single biggest lever for improving translation quality in low-resource, morphologically rich languages.
We present BhashaSetu, a linguistically enriched English--Marathi parallel dataset addressing persistent data limitations in low-resource neural machine translation (NMT). Marathi, spoken by over 95 million people, remains underrepresented in high-quality parallel corpora across diverse domains. Our dataset comprises 2.78 million sentence pairs from heterogeneous sources including news, politics, healthcare, literature, and culture, with stemmed and lemmatized representations to support morphology-aware analysis. We benchmark multiple state-of-the-art translation models using BLEU, spBLEU, chrF++, and TER metrics, and conduct parameter-efficient fine-tuning of NLLB-200-distilled-600M using LoRA. A key finding from our ablation: corpus-level deduplication is the single largest preprocessing contributor to downstream quality (removing it reduces performance by 1.17 BLEU and 2.21 chrF++), demonstrating that disciplined cross-source corpus hygiene is a low-cost, high-impact intervention for low-resource, morphologically rich languages. The dataset is publicly released to promote reproducible and linguistically informed low-resource NMT research.