Search papers, labs, and topics across Lattice.
This paper introduces HCRMap, a framework designed to optimize the residency of hot expert replicas in 3.5D Mixture-of-Experts (MoE) systems, addressing the issue of expert hotness skew that leads to resource bottlenecks during inference. By dynamically managing expert placements based on hotness, weight loading costs, and runtime pressures, HCRMap significantly improves the efficiency of token routing and resource utilization. Experimental results demonstrate that HCRMap achieves a remarkable reduction in end-to-end latency, outperforming existing methods like Hydra and MoEntwine by over 43% in various stages of inference.
HCRMap slashes end-to-end latency by up to 46.7% in 3.5D MoE inference by intelligently managing expert residency based on real-time pressure metrics.
Mixture-of-Experts (MoE) large language models (LLM) activate only a small number of experts during inference, but token routing introduces persistent expert hotness skew: a small set of hot experts continuously receives most tokens, while the remaining experts are lightly loaded. On 3.5D multi-chiplet systems, this skew not only causes compute imbalance but also amplifies pressure on communication, memory bandwidth, I/O, and execution queues. Therefore, the core problem is not simply to reduce token movement, but to dynamically place and reuse hot expert replicas across different memory tiers. This paper proposes HCRMap, a hot expert residency mapping framework for pressure-aware expert replica management in 3.5D MoE inference. Based on expert hotness, weight loading cost, migration overhead, and runtime resource pressure, HCRMap dynamically determines which experts should be promoted, retained, demoted, or evicted. It then maps routed token groups to suitable resident replicas, thereby jointly mitigating communication, memory, and queue bottlenecks. Experimental results show that HCRMap reduces end-to-end latency by 43.6% and 43.0% over Hydra in the prefill and decode stages, respectively; by 34.5% and 33.1% over MoEntwine; and by 46.7% and 46.0% over PIMoE.
