Search papers, labs, and topics across Lattice.
This paper introduces MOSAIC, a framework that integrates model architecture and systems co-design for optimizing sparse Mixture-of-Experts (MoE) language models. By coupling a predictive scaling law with a calibrated performance model, MOSAIC evaluates Model FLOPs Utilization (MFU), communication costs, and memory footprints, leading to insights on optimal sparsity levels that enhance both loss and systems efficiency. The findings reveal that the compute-optimal model configuration does not align with the cluster's systems constraints, advocating for a unified approach in training large-scale language models.
Optimal sparsity in sparse MoE models emerges only when considering the cluster's systems constraints, challenging traditional compute-centric design paradigms.
In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages. A scaling law stage selects an architecture and training recipe, optimizing loss under compute constraints, and a separate systems stage then optimizes the implementation for hardware efficiency. In this work, we develop MOSAIC, which formulates model architecture and systems co-design as an optimization problem. MOSAIC couples a predictive scaling law with a calibrated performance model that estimates Model FLOPs Utilization (MFU), communication cost, memory footprint, and the best parallel layout. We instantiate the framework for sparse Mixture-of-Experts (MoE) language models, where expert count, routing sparsity, and other MoE layer dimensions affect both the loss and systems efficiency. We fit a scaling law on sparse MoE models trained on text data, whose scaling dimensions include the sparsity factor, which is the fraction of model parameters inactive per token in a forward pass. The scaling law sweeps in our work span active parameters from $104$ million to $2.7$ billion and total model sizes reaching $79$ billion parameters. We show that, within the calibrated sparsity range, an efficiency-agnostic model-FLOPs budget admits no interior optimal sparsity. The fitted loss decreases monotonically with sparser models and the compute optimum lies at the upper boundary of the data support. An optimal sparsity in MoE models instead emerges under the cluster's systems constraints, as captured by MOSAIC. Our results argue for a shift towards unified architecture and systems co-design for frontier language model training.
