Search papers, labs, and topics across Lattice.
The paper introduces Llamion, a family of 14B-parameter language models derived from Orion-14B by converting it to the Llama architecture. The conversion leverages a novel recipe called KEPT, combining Normal Parameter Mapping, Optimized Parameter Mapping (a training-free LayerNorm-to-RMSNorm initialization), and Cross-architecture Knowledge Distillation. Llamion achieves strong performance, surpassing existing open KoLLMs on KoMMLU by over 7 points, while preserving capabilities like Python programming and 200K-token context handling, despite limited fine-tuning.
Llamion proves you can transplant a model's brain (capabilities like long context and coding) into a new body (Llama architecture) with minimal training, opening the door to easier architecture upgrades and model merging.
We release Llamion, a family of 14B-parameter open-weight language models obtained by transforming Orion-14B into the standardized Llama-family architecture. The transformation is performed by Efficient Knowledge Preservation for Transformation (KEPT), a recipe that combines (i) Normal Parameter Mapping (NPM) for unchanged modules, (ii) Optimized Parameter Mapping (OPM), a training-free LayerNorm-to-RMSNorm initialization we prove optimal under the near-zero-mean activation regime induced by weight decay, and (iii) Cross-architecture Knowledge Distillation (XKD), an equal-size frozen-teacher distillation that aligns the converted model's outputs with the source model's on any reasonable input distribution. Llamion recovers Orion's behaviour on H6, MT-Bench, and KoMMLU with only ~123M tokens on a single A100 in four days; Llamion-Base reaches 66.87% on KoMMLU, exceeding the next-best entry of the Open Ko LLM Leaderboard by >7.0 absolute points at submission time. Capabilities entirely absent from the transfer corpus (Python programming and 200K-token context handling) survive the architectural transition intact. We release three checkpoints (Base, Chat, LongChat) that load with trust_remote_code=False in the Hugging Face Transformers library.
