Search papers, labs, and topics across Lattice.
This paper introduces LLMET, a validated cross-layer simulation framework that evaluates the impact of emerging monolithic 3D (M3D) memory technologies on energy efficiency in Large Language Model (LLM) serving. By expanding on-chip memory capacity, the study demonstrates significant reductions in chip energy consumption during critical phases of model operation, achieving up to 44% energy savings with a 1GB L2 cache in the Llama3.1-70B model. The findings indicate that leveraging ultra-large on-chip memories can substantially mitigate the energy challenges associated with LLM deployment across various platforms and workloads.
Expanding on-chip memory to 1GB can slash energy consumption by 44% during LLM prefill phases, revealing a game-changing approach to LLM efficiency.
The energy consumption of Large Language Model (LLM) serving is becoming a major system challenge as deployment scales, driven by hardware power and thermal constraints and rising electricity costs. A key contributor to chip energy dissipation is data movement between limited on-chip cache and off-chip High Bandwidth Memory (HBM). Meanwhile, emerging memory technologies such as monolithic 3D (M3D) integration of cache memories at the Back-End-Of-Line (BEOL) of logic chips enable larger and denser on-chip memories, creating new opportunities to reduce costly off-chip traffic. However, it remains unclear whether continuously scaling on-chip memory using emerging technologies can effectively improve the energy efficiency of LLM serving. To address this gap, we develop LLMET (LLM with Emerging Technology), a validated cross-layer simulation framework, and conduct a comprehensive study on the impact of large-capacity on-chip memory technologies across a broad range of models, applications and platforms. Utilizing M3D technology to expand the L2 cache from 40MB to 1GB yields a 44% reduction in chip energy during the Llama3.1-70B prefill phase with a 16K context window, based on LLMET simulation on a dual NVIDIA A100 GPU setup. On the 8x NVIDIA B200-like platform, extending the L2 cache from 128MB to 4GB saves the prefill energy by up to 24%. For the edge platform and workloads, the decode energy saving reaches 30% when increasing the 8MB cache size to 256MB. These results highlight the promise of ultra-large on-chip memories for energy-efficient LLM serving systems.
