Search papers, labs, and topics across Lattice.
This paper introduces OrderMoE, a similarity-aware framework for deploying mixture-of-experts (MoE) models in resource-constrained edge environments, addressing the challenges of inference latency and communication overhead. By leveraging expert similarity to optimize expert allocation and grouping, OrderMoE minimizes cross-server token transmission and balances server workloads while maintaining inference quality. Experimental results demonstrate that OrderMoE significantly reduces average and tail latency, as well as remote expert invocation, with only minor quality degradation, showcasing its effectiveness in edge deployments.
OrderMoE slashes average latency and cross-server traffic by intelligently grouping experts based on functional similarity, achieving efficient edge inference without sacrificing quality.
Although mixture-of-experts, MoE, models have been increasingly adopted to scale large language models with moderate computation cost, it remains challenging to deploy MoE inference over resource-constrained and bandwidth-limited edge infrastructures. Existing distributed MoE serving methods mainly rely on exact expert placement, caching, replication, or communication scheduling, while overlooking the functional similarity among experts, which provides an opportunity to reduce cross-server token transmission. Therefore, this paper introduces a similarity-aware expert allocation and distributed deployment framework, dubbed OrderMoE, which aims to accelerate edge MoE inference while balancing inference latency, communication overhead, server workload, and inference quality. OrderMoE first constructs an expert similarity model based on router-induced logits representations and partitions experts in each MoE layer into multiple similarity groups. Then, it develops a similarity-aware expert grouping and deployment strategy to improve local similarity coverage across edge servers. Since reducing remote expert invocation and preserving exact inference quality are conflicting objectives, OrderMoE further designs a quality-aware and trajectory-aware runtime server-expert selection algorithm to decide whether a token should invoke its remote target expert or use a feasible local substitute expert. Experimental results on a real distributed edge testbed show that OrderMoE significantly reduces average latency, tail latency, cross-server traffic, and remote expert invocation ratio, while introducing only small and controllable inference quality degradation.
