Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces FastEEP, a novel system for intra-instance elastic scaling of expert-parallel Mixture-of-Experts (MoE) models during serving. By employing a "Partial Reuse" principle that separates reusable execution structures from activated resources, FastEEP avoids the costly destruction and reconstruction of entire serving instances inherent in traditional instance-level scaling. Evaluations demonstrate that FastEEP achieves significantly faster scaling (4.5x-11.7x) compared to existing methods, enabling more efficient and fine-grained capacity adjustments for dynamic MoE workloads.
MoE serving can now scale 4.5x-11.7x faster by reusing existing runtime state, sidestepping costly instance reconstruction.
Large-scale expert parallelism (EP) has become the standard deployment configuration for Mixture-of-Experts (MoE) models in production. Meanwhile, the highly dynamic nature of LLM workloads requires providers to continuously adjust serving capacity to match fluctuating demand. Scaling MoE serving cannot be efficiently achieved by simply adding or removing serving instances; instead, it often requires resizing the EP group within an existing instance, which we refer to as intra-instance scaling.Existing inference engines primarily rely on instance-level scaling, which destroys and reconstructs entire serving instances, incurring substantial scaling overhead. We present FastEEP (Fast Elastic Expert Parallel), a fast intra-instance scaling system for MoE serving based on the principle of Partial Reuse. FastEEP reuses existing runtime state across worker execution, GPU memory, and communication by separating reusable execution structures from the resources activated by the current EP configuration. This separation enables coordinated resource reclamation and reconfiguration while preserving costly-to-reconstruct state, allowing fast EP scaling without full instance teardown and reconstruction. Our evaluation shows that FastEEP achieves 4.5x--11.7x faster scaling than the existing vLLM-based approach across different models, enabling efficient, fine-grained scaling of MoE serving.