Search papers, labs, and topics across Lattice.
DynaMoE is introduced, a novel Mixture-of-Experts (MoE) framework that allows dynamic token-level expert activation and layer-wise adaptive capacity allocation, relaxing the constraints of fixed Top-K routing and uniform expert allocation. The framework employs a routing mechanism where the number of active experts per token varies based on input complexity, and implements six scheduling strategies for distributing expert capacity across network depth. Experiments across image classification and language modeling tasks demonstrate that DynaMoE achieves superior parameter efficiency compared to static baselines, with optimal expert schedules being task- and scale-dependent.
Forget fixed routing: DynaMoE dynamically adjusts expert activation per token and layer, unlocking better parameter efficiency and convergence stability in Mixture-of-Experts models.
Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling neural networks while maintaining computational efficiency. However, standard MoE implementations rely on two rigid design assumptions: (1) fixed Top-K routing where exactly K experts are activated per token, and (2) uniform expert allocation across all layers. This paper introduces DynaMoE, a novel MoE framework that relaxes both constraints through dynamic token-level expert activation and layer-wise adaptive capacity allocation. DynaMoE introduces a principled routing mechanism where the number of active experts per token varies based on input complexity. Concurrently, the framework implements six distinct scheduling strategies for distributing expert capacity across network depth, including descending, ascending, pyramid, and wave patterns. We theoretically analyze the expressivity gains of dynamic routing and derive bounds on computational efficiency. Through extensive experiments on MNIST, Fashion-MNIST, CIFAR-10 (image classification), and Recycling-the-Web (language modeling) across multiple model scales, we demonstrate that DynaMoE achieves superior parameter efficiency compared to static baselines. Our key finding is that optimal expert schedules are task- and scale-dependent: descending schedules (concentrating capacity in early layers) outperform uniform baselines on image classification. For language modeling, optimal schedules vary by model size, descending for Tiny, ascending for Small, and uniform for Medium. Furthermore, dynamic routing reduces gradient variance during training, leading to improved convergence stability. DynaMoE establishes a new framework for adaptive computation in neural networks, providing principled guidance for MoE architecture design.
