Search papers, labs, and topics across Lattice.
This paper introduces ThAME, a novel 3D heterogeneous multi-chiplet architecture designed to optimize Mixture of Experts (MoE) inference for Large Language Models (LLMs) by addressing critical bottlenecks in memory bandwidth, token routing, and output aggregation. By integrating Ferroelectric Field-Effect Transistor (FeFET) and DRAM-based memory chiplets with a tailored compute mapping strategy, ThAME significantly enhances the efficiency of expert routing and communication. Experimental results reveal that ThAME achieves up to 15.7x speedup and 9.8x energy efficiency improvement compared to existing solutions, marking a substantial advancement in MoE architecture performance.
ThAME achieves a staggering 15.7x speedup in MoE inference, revolutionizing the efficiency of Large Language Models.
Mixture of Experts (MoE) architectures have emerged as a dominant paradigm for scaling Large Language Models (LLMs). However, MoE inference on conventional hardware is constrained by three fundamental bottlenecks. These encompass the massive memory bandwidth required to fetch non-contiguous expert weights, the non-deterministic scatter-gather traffic generated by input-dependent token routing, and the tail-latency dependency imposed by synchronous expert output aggregation. To address these challenges, we propose ThAME, a three-dimensional (3D) heterogeneous multi-chiplet architecture for MoE inference. ThAME employs Ferroelectric Field-Effect Transistor (FeFET)-based non-volatile and DRAM-based volatile memory chiplets with a co-designed compute mapping strategy that aligns the distinct computational profiles of attention mechanisms and expert routing. Furthermore, we design a specialized Network-on-Chip communication backbone optimized to mitigate the bottlenecks associated with non-deterministic token routing traffic across the combinatorial space of input-dependent MoE traffic patterns. Experimental results demonstrate that ThAME outperforms state-of-the-art counterparts by up to 15.7x in terms of speedup and improves energy efficiency by up to 9.8x.
