Search papers, labs, and topics across Lattice.
This paper introduces ASAP, an asynchronous inference system designed to enhance the prefill phase of Mixture-of-Experts (MoE) models by disaggregating attention and MoE stages. The motivation stems from the inefficiencies caused by global synchronization barriers in current hybrid parallelism strategies, which lead to significant stalls and degraded performance in online serving scenarios. ASAP employs specialized asynchronous communication primitives and optimizations, resulting in a remarkable 90% improvement in SLO-compliant prefill throughput over existing synchronous solutions.
Achieving a 90% boost in prefill throughput for MoE models could redefine the efficiency of large-scale language model serving.
Mixture-of-Experts (MoE) models have become the de facto standard for scaling large language models. To maintain computational efficiency, modern MoE serving systems typically employ a hybrid parallelism strategy, combining Data Parallelism (DP) for attention stages with Expert Parallelism (EP) for MoE stages. However, this design necessitates frequent global synchronization barriers between attention DP groups and experts. In online serving, significant variance in request arrival rates and sequence lengths inherently leads to DP imbalance, causing severe synchronization stalls that degrade Time-to-First-Token (TTFT) and system throughput. We present ASAP, an asynchronous inference system specifically designed to accelerate the prefill phase of MoE models. ASAP disaggregates the attention and MoE stages and implements a fully asynchronous execution pipeline. This is achieved through a suite of specialized asynchronous communication primitives and four coordinated optimizations across request scheduling and model execution, which collectively dismantle global synchronization barriers. We implement and evaluate ASAP on CloudMatrix384 super-nodes, demonstrating that it improves SLO-compliant prefill throughput by 90% compared to state-of-the-art synchronous serving solutions.
