Search papers, labs, and topics across Lattice.
This paper addresses the scarcity of pretraining data for Armenian by curating and releasing two comprehensive datasets: ArmWeb, a corpus of 4.37M Armenian news documents, and ArmSTEM, a parallel English-Armenian collection of 373K math and science problems. The researchers then continued pretraining the Gemma-4-E4B model on these datasets, resulting in arm-gemma-e4b, which surpasses all existing open Armenian models and is the first to provide complete training data and methodology. Their findings reveal that while news-only pretraining enhances fluency, it diminishes knowledge retention, a trend mitigated by incorporating a small amount of verified STEM data.
The first open Armenian LLM, arm-gemma-e4b, not only outperforms all predecessors but also highlights the critical balance between fluency and knowledge retention in low-resource language models.
Pretraining data for Armenian, a morphologically rich and low-resource language, is scarce, and no open Armenian LLM has been released with the data and recipe needed to reproduce it. To address this gap, we curate and release two datasets. ArmWeb is an extensively validated corpus of 4.37M Armenian news documents. ArmSTEM is a parallel English-Armenian collection of 373K math and science problems with step-by-step solutions, translated into Armenian and verified through both answer-preserving LLM judgment and human evaluation. Continued pretraining of Gemma-4-E4B on these datasets yields arm-gemma-e4b, which outperforms every existing open Armenian model as well as its unadapted base, and is the first open Armenian LLM with complete training data and recipe. Our ablations show that news-only continued pretraining improves fluency while eroding knowledge, a pattern we also observe in existing Armenian models, and that a small share of verified translated STEM data reverses the loss. We further find that the largest public Armenian corpora overlap web-derived evaluation panels heavily, including a train/test self-overlap inside FineWeb-2. We openly release all data, models, and code.
