Search papers, labs, and topics across Lattice.
This paper introduces Moebius, a serving system for Mixture-of-Experts (MoE) models that dynamically switches between tensor parallelism (TP) and expert parallelism (EP) at runtime, optimizing performance based on workload concurrency. By leveraging the fact that both parallelism layouts compute the same function over identical weights and KV cache, Moebius enables seamless transitions without the need to restart or drop in-flight requests. The results demonstrate that Moebius not only matches the performance of the best static parallelism configuration but also outperforms it during reinforcement learning rollouts by 1.16-1.25x, with minimal memory overhead.
Moebius achieves up to 1.25x better performance in reinforcement learning rollouts by seamlessly switching between expert and tensor parallelism without disrupting in-flight requests.
Mixture-of-Experts (MoE) architectures scale large language models (LLMs) to hundreds of billions of parameters. Serving a single MoE model requires multiple GPUs operating in parallel, typically through tensor parallelism (TP) or expert parallelism (EP). The optimal choice depends on the number of in-flight requests: TP is faster at low concurrency, whereas EP wins at high concurrency. Production workloads cross this boundary continually: online serving sees bursty arrivals that subside into quiet periods, and reinforcement-learning rollouts begin as a high-concurrency burst that decays into a long tail of stragglers. Pinning either layout therefore forfeits performance when the workload crosses to the other side. We present Moebius, a serving system that switches between EP and TP at runtime without restarting the engine or dropping in-flight requests. Our key insight is that EP and TP are two layouts of one model, not two models: they compute the same function over byte-identical expert weights and KV cache, so a switch changes only which rank owns each slice. Moving those owner-changed slices is the sole irreducible cost, and modern high-bandwidth GPU interconnects make it fast enough to do between decode steps without draining in-flight requests. Moebius preserves each parallelism's runtime resident, and reshards the single copy of expert weights and KV cache at fixed addresses with fused GPU-to-GPU transfer kernels. On 8x H200 GPUs serving Qwen3-235B-A22B, Moebius matches the better static parallelism at every operating point, and beats it on RL rollouts by 1.16-1.25x across steps. Each switch completes in 215-434 ms, and Moebius holds both layouts resident with only 2.4% memory overhead.
