Andreea Tomescu

L−1log⁡pS​(i,yi+1),\mathcal{L}_{\text{CE}}=-\sum_{i=1}^{L-1}\log p_{S}(i,y_{i+1}), where LL denotes the length of the tokenized sequence, and yi+1y_{i+1} is the ground-truth token at position i+1i{+}1 in the sequence. Training follows the standard autoregressive formulation: at step ii, the student predicts the next token conditioned on the prefix {y1,…,yi}\{y_{1},\dots,y_{i}\}. Padding positions are excluded from the loss via ignore_index. This prevents degenerate behavior when the student initially differs substantially from the teacher. Loss Weighting. TF3 uses α=1.0\alpha=1.0 and β=0.1\beta=0.1, prioritizing imitation while retaining a ground-truth anchor. 7.3 Training Procedure Knowledge distillation in TF3 is performed on the same Romanian corpus used for teacher pretraining, with identical tokenization to ensure full vocabulary alignment between models. Training retains the long-sequence packing strategy described in Section 6, preserving the positional distributions encountered during from-scratch pretraining and avoiding distributional shifts during compression. The student model is optimized using the AdamW optimizer with weight decay [34], combined with mixed-precision computation and gradient accumulation to match the teacher’s effective batch size under hardware constraints. Both teacher and student operate on packed sequences of 2,048 tokens, and distillation proceeds for several epochs until convergence. During training, the teacher remains frozen and provides soft target distributions, while the student performs forward and backward passes only, making the procedure substantially more efficient than retraining a reduced architecture from scratch. 7.4 Model Export and Deployment After convergence, the distilled student model and its tokenizer are exported in a Hugging Face–compatible format under artifacts/tf3-student-26m. With approximately 26., ×\times for comparable text lengths [45]. This so-called “tokenization penalty” restricts usable context, increases computational cost, and biases training toward English-centric segmentation patterns. As a consequence, Romanian users face higher inference latency, reduced context utilization, and limited access to compact, locally deployable language models. The TinyFabulist project series was initiated to address this structural gap by combining controlled synthetic data generation with linguistically informed modeling pipelines. TF1 introduced a three-million-story English dataset of moral microfiction generated via a structured six-slot narrative scaffold, demonstrating that pedagogically constrained narratives can yield coherent and diverse text at scale. Building on this foundation, TF2 extended the dataset with high-quality Romanian translations and released the first large-scale bilingual synthetic corpus of moral fables via Hugging Face (ds-tf2-en-ro-3m) [37, 38]. Together, TF1 and TF2 established synthetic narrative generation as a viable data source for Romanian language modeling. Nevertheless, what remained missing was a complete, reproducible recipe for building Romanian language models end-to-end. While prior work addressed individual components in isolation, no unified pipeline existed that begins with synthetic narrative data; constructs Romanian-specific tokenizers; produces stable, long-context training sequences; trains compact models from scratch; evaluates them using linguistically grounded benchmarks; compresses them via quantization, pruning, and knowledge distillation; and ultimately uses the resulting models to generate new large-scale synthetic datasets. This end-to-end perspective is especially critical for Romanian deployments, where practical constraints require clear answers to a set of interrelated questions: (i) how to mitigate the token inflation induced by Romanian morphology; (ii) how to train compact Romanian models from scratch using exclusively synthetic corpora; (iii) how to evaluate such models along grammatical, narrative, and efficiency dimensions; and (iv) how to compress them for local deployment without incurring major quality degradation. In response to these challenges, we introduce TF3, the first pipeline that completes this trajectory for Romanian language modeling. TF3 begins with the Romanian split of TF2 and constructs dedicated BPE and Unigram tokenizers explicitly optimized for Romanian morphology. These tokenizers enable efficient packing of the corpus into 2,048-token blocks, yielding an approximately one-billion-token training set. On top of this corpus, TF3 trains a 51.