Search papers, labs, and topics across Lattice.
The Qwen3.8-Flash-Next architecture is a sparse mixture-of-experts model with 125 billion parameters, optimized for efficiency and training stability through innovative token mixing and n-gram embedding strategies. It outperforms its 397B-A17B predecessor on eight out of fourteen pre-training benchmarks while utilizing only one-third of the activated parameters and training tokens, and achieving roughly one-ninth the training FLOPs. Key design features include a hybrid of Gated DeltaNet and global attention, along with a Gated Residual mechanism that enhances stability and performance during training.
Achieving superior performance with one-third the resources, Qwen3.8-Flash-Next redefines efficiency in large-scale language models.
We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and additional 51B parameters of n-gram embedding tables held off the accelerator. On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs. Token mixing uses a layer-wise hybrid of Gated DeltaNet (GDN) and global attention, with one full-attention layer in every four; at continued-pretraining time those full-attention layers are replaced by Qwen Sparse Attention (QSA), which scores context at micro-block granularity with a compressed lightweight indexer. The residual stream is widened to four branches and read through an elementwise gate, a design we call the Gated Residual (GR). Capacity is added outside the backbone by a single n-gram embedding layer whose tables are prefetched from host memory. We evaluate every candidate change along three axes: loss together with downstream benchmarks; the cost of the change in training, prefill and decode; and its effect on the optimal hyperparameters and training stability. Loss and downstream accuracy do not always move together: enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates. The architecture and the Muon optimizer together shift the optimal learning rate and batch size upwards, render batch-size warmup unnecessary, and substantially improve stability under stress tests. Loss, benchmarks, efficiency and stability form one design problem. Solved jointly, they yield a recipe that is simultaneously more efficient, more capable and more stable.
