Search papers, labs, and topics across Lattice.
This paper introduces Expert Tying, an architectural innovation that shares parameters of experts across consecutive transformer layers in Mixture-of-Experts (MoE) language models while maintaining independent routing and attention mechanisms. The approach significantly reduces the memory footprint by nearly 2x without compromising perplexity or downstream performance across various state-of-the-art architectures, such as OLMoE and Qwen3. This advancement leverages the inherent parameter redundancy in MoE pathways, enhancing the efficiency of training and scaling large language models.
Tying expert parameters across layers can halve memory usage in MoE models without sacrificing performance, revolutionizing how we scale LLMs.
Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference memory. To address this, we introduce Expert Tying, an architectural modification that shares expert parameters across consecutive transformer layers while preserving independent, layer-wise routing and attention. We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs. Our pretraining experiments demonstrate that tying experts can reduce memory footprint by almost 2x at virtually no degradation in perplexity or downstream quality. By exploiting the parameter redundancy inherent in MoE pathways, our method provides a highly favorable compute-to-memory trade-off, advancing efficient training and scaling of next-generation LLMs.
