Search papers, labs, and topics across Lattice.
The paper introduces Turing-20B-A2B, a 20B-parameter Mixture-of-Experts language model that activates around 2B parameters per token, specifically tailored for long-context and latency-sensitive physical AI applications. By employing Quantile Routing in a dynamic top-k setup and a hybrid attention architecture, the model achieves a remarkable balance between expert utilization and compute efficiency, allowing for an extended context length of 128K, and even 512K during inference. Notably, Turing-20B-A2B outperforms the Qwen3-8B Base model while maintaining superior long-context performance and efficient latency scaling.
Turing-20B-A2B achieves exceptional long-context capabilities with only a fraction of its parameters activated, pushing the boundaries of efficiency in physical AI applications.
We present Turing-20B-A2B, a 20B-parameter Mixture-of-Experts language model that activates approximately 2B parameters per token, designed for long-context and latency-sensitive physical AI applications. The model adopts Quantile Routing in a dynamic top-k configuration, enabling token-adaptive expert allocation while maintaining balanced expert utilization and a controlled average compute budget. During deployment, we further apply capacity-constrained routing to prompt prefill for more regular and efficient expert execution, while retaining dropless routing during pretraining. Turing-20B-A2B also employs a hybrid attention architecture that combines Lightning Attention with a small number of full-attention layers for efficient long-context modeling. The model is pretrained with a progressive three-stage curriculum and extended to a native context length of 128K through continued pretraining, with further inference-time extension to 512K using YaRN. Despite its compact active-parameter budget, Turing-20B-A2B achieves, at the base-model stage, overall general capability exceeding Qwen3-8B Base and approaching Qwen3.5-9B Base, while maintaining strong long-context performance and favorable prefill-latency scaling. These results demonstrate an effective balance among model capability, long-context scalability, and practical inference efficiency.
