Search papers, labs, and topics across Lattice.
This technical survey analyzes the evolution of Mixture-of-Experts (MoE) architectures in large language models by organizing key developments along five interrelated dimensions: expert granularity, expert topology, routing freedom, load balancing, and execution structure. By presenting a dependency graph of eight architectural milestones, the authors highlight a significant shift from simply activating sparse parameters to a more sophisticated decoupling of semantic routing and computational resources. The study culminates in equal-budget pretraining experiments that reveal the impact of these architectural choices on model quality and system performance, while also identifying open research questions in the field.
The evolution of Mixture-of-Experts architectures reveals a critical shift towards decoupling semantic routing from computational budgets, reshaping our understanding of model efficiency.
Mixture-of-Experts models increase parameter capacity while keeping the computation activated by each token bounded, but their architectural evolution cannot be explained by a chronological list of model releases alone. This technical survey synthesizes primary papers, official technical reports, and prior surveys to organize modern Mixture-of-Experts systems along five coupled dimensions: expert granularity, expert topology, routing freedom, the scope of load balancing, and execution structure. We describe eight architectural milestones as a dependency graph with six mainline developments and two orthogonal branches, rather than as eight successive generations. We then analyze individual systems through four control planes: Expert Topology, Routing, Balance, and Expert Parallelism. These planes specify which experts exist, which experts process each token, how aggregate load is controlled, and how selected computation is mapped onto physical devices. The framework connects algorithmic choices such as Top-k routing, shared experts, fine-grained experts, and dynamic expert composition with systems concerns including token dispatch, device placement, all-to-all communication, and communication-computation overlap. We conclude with equal-budget pretraining experiments, quality and systems metrics, and open research questions. The main trend is a shift from merely activating more sparse parameters toward decoupling semantic routing, computational budgets, and physical execution.
