Search papers, labs, and topics across Lattice.
The paper introduces ExpertWeaver, a training-free framework for converting dense language models into Mixture-of-Experts (MoE) architectures by leveraging the activation patterns within Gated Linear Units (GLU). It argues that GLU activation patterns reveal an inherent MoE structure of universal and specialized neurons. ExpertWeaver partitions neurons based on their activation patterns to construct shared and routed experts, achieving superior performance compared to existing dense-to-MoE conversion methods.
Forget expensive MoE training from scratch: ExpertWeaver unlocks inherent MoE structure within dense LLMs using GLU activation patterns, offering a training-free conversion.
Mixture-of-Experts (MoE) effectively scales model capacity while preserving computational efficiency through sparse expert activation. However, training high-quality MoEs from scratch is prohibitively expensive. A promising alternative is to convert pretrained dense models into sparse MoEs. Existing dense-to-MoE methods fall into two categories: \textbf{dynamic structural pruning} that converts dense models into MoE architectures with moderate sparsity to balance performance and inference efficiency, and \textbf{downcycling} approaches that use pretrained dense models to initialize highly sparse MoE architectures. However, existing methods break the intrinsic activation patterns within dense models, leading to suboptimal expert construction. In this work, we argue that the Gated Linear Unit (GLU) mechanism provides a natural blueprint for dense-to-MoE conversion. We show that the fine-grained neural-wise activation patterns of GLU reveal a coarse-grained structure, uncovering an inherent MoE architecture composed of consistently activated universal neurons and dynamically activated specialized neurons. Leveraging this discovery, we introduce ExpertWeaver, a training-free framework that partitions neurons according to their activation patterns and constructs shared experts and specialized routed experts with layer-adaptive configurations. Our experiments demonstrate that ExpertWeaver significantly outperforms existing methods, both as a training-free dynamic structural pruning technique and as a downcycling strategy for superior MoE initialization.
