Search papers, labs, and topics across Lattice.
This paper introduces a novel compute microarchitecture optimized for 3D-stacked near-memory processing (NMP) to accelerate LLM decoding, addressing the challenge of compute-bound operations arising from the high memory bandwidth of 3D-stacked NMP. The design replaces MAC tree-based compute units with an area-efficient and reconfigurable systolic array, leveraging existing vector cores for control logic and buffering. Evaluations show a 2.91x speedup and 2.40x higher energy efficiency compared to Stratum across dense and MoE models, highlighting the benefits of the proposed multi-core scheduling framework.
LLM decoding on 3D-stacked memory shifts the bottleneck from memory bandwidth to compute, demanding a rethink of compute microarchitecture.
Large language model (LLM) decoding is a major inference bottleneck because its low arithmetic intensity makes performance highly sensitive to memory bandwidth. 3D-stacked near-memory processing (NMP) provides substantially higher local memory bandwidth than conventional off-chip interfaces, making it a promising substrate for decode acceleration. However, our analysis shows that this bandwidth advantage also shifts many decode operators on 3D-stacked NMP back into the compute-bound regime. Under the tight area budget of the logic die, the design of the compute substrate itself therefore becomes a first-order challenge. Therefore, we rethink the compute microarchitecture of prior 3D-stacked NMP designs. First, we replace prior MAC tree-based compute units with a more area-efficient systolic array, and we further observe that decode operators exhibit substantial shape diversity, making reconfigurability in both systolic array shape and dataflow essential for sustaining high utilization. Building on this insight, we continue to exploit two key opportunities: the high local memory bandwidth reduces the need for large on-chip buffers, and the existing vector core, originally designed to handle auxiliary tensor computations, already provides much of the control logic and multi-ported buffering required for fine-grained flexibility for systolic array, allowing us to unify the two structures in a highly area-efficient manner. Based on these insights, we present the first compute microarchitecture tailored to 3D-stacked NMP LLM decoding, explicitly designed to satisfy the joint requirements of low area cost, high-bandwidth operation, and fine-grained reconfigurability. We further propose an multi-core scheduling framework. Compared with Stratum, our design achieves an average 2.91x speedup and 2.40x higher energy efficiency across both dense and MoE models.
