Search papers, labs, and topics across Lattice.
Instella-MoE is a fully open Mixture-of-Experts language model featuring 16 billion total parameters, with 2.8 billion active parameters per token, trained from scratch on advanced AMD GPUs. The model integrates innovative architectural designs such as Gated Multi-head Latent Attention and FarSkip-Collective connectivity, facilitating efficient large-scale training and inference. Achieving an average score of 76.7 on pre-training benchmarks, Instella-MoE surpasses previous open models while maintaining competitiveness with dense baselines, and it supports reproducible research through comprehensive model release.
Outperforming prior open models, Instella-MoE sets a new standard for efficiency and performance in language modeling with its innovative architecture and training methods.
In this work, we introduce Instella-MoE, a fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters per token, trained entirely from scratch on AMD Instinct MI300X and MI325X GPUs. Instella-MoE combines a sparsely activated MoE design with architectural and system-level innovations, including Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective connectivity, enabling efficient large-scale training and inference. The model is developed through a multi-stage pipeline comprising pre-training, mid-training, long-context extension, supervised fine-tuning with feedback-driven data curation, direct preference optimization, and reinforcement learning with Multi-Teacher On-Policy Distillation. Instella-MoE achieves an average score of 76.7 across standard pre-training benchmarks, outperforming prior fully open models including OLMo-3-7B, SmolLM3-3B, and OLMoE-1B-7B, while remaining competitive with open-weight MoE and dense baselines at comparable active-parameter scales, including Moonlight-16B-A3B and Qwen3.5-4B. After post-training, our final Think checkpoint achieves an average score of 73.2 across instruction-following, reasoning, math, coding, and chat benchmarks, outperforming both fully open and open-weight models with comparable or larger active parameter counts in our evaluation. To support transparent and reproducible research, we release the complete Instella-MoE model flow, including model weights, training configurations, data mixtures, and training code. Together, these contributions establish Instella-MoE a strong, fully open foundation for efficient, high-performing MoE models and reproducible research.
