Search papers, labs, and topics across Lattice.
This paper introduces TF3-RO, a comprehensive pipeline for training compact Romanian language models from scratch, addressing the lack of such resources for morphologically rich, under-resourced languages. The pipeline encompasses tokenizer design, pretraining of a 51.65M-parameter LLaMA-style Transformer, compression via quantization/pruning/distillation, and the generation of a 3-million story Romanian fable dataset. The resulting 26.45M-parameter distilled model demonstrates strong deployment characteristics and facilitates the creation of large-scale synthetic narrative corpora, evaluated using a suite of intrinsic and extrinsic metrics.
A complete, reproducible pipeline now exists for training compact, high-quality Romanian language models from scratch, leveraging synthetic data and careful tokenizer design to overcome resource limitations.
Recent advances in synthetic data generation have shown that compact language models can be trained effectively when the underlying corpus is structurally controlled and linguistically coherent. However, for morphologically rich and computationally under-resourced languages such as Romanian, there is still no openly documented, end-to-end pipeline that unifies tokenizer design, preprocessing, pretraining, compression, evaluation, and large-scale synthetic data generation in a reproducible framework. Building on TF1, a three-million-story English fable dataset, and TF2, which extends TF1 through high-quality Romanian translations, we introduce TF3-RO, a Romanian-centric language modeling pipeline spanning tokenizer training, from-scratch model development, and Romanian-native dataset generation. TF3-RO constructs Romanian-specific BPE and Unigram tokenizers from a linguistically informed corpus to mitigate token inflation induced by Romanian morphology. Using long-sequence packed training, we pretrain a 51.65M-parameter LLaMA-style Transformer entirely from scratch. The model is subsequently optimized through quantization, structured pruning, and logit-based knowledge distillation, yielding a compact 26.45M-parameter student model with tied embeddings and strong deployment characteristics. Using this distilled model, TF3-RO generates three million Romanian-native synthetic fables via a controlled combinatorial prompting framework. Across all stages, the pipeline integrates a comprehensive evaluation suite combining intrinsic metrics, Romanian agreement probes, entity coherence, rule-based grammar checking, and LLM-based assessment. TF3-RO provides a reproducible and linguistically grounded framework for training compact Romanian language models and producing large-scale synthetic narrative corpora.
