Search papers, labs, and topics across Lattice.
This paper evaluates the efficiency of heterogeneous DRAM-based processing-in-memory (PIM)-GPU systems for large language model (LLM) inference, revealing that static power consumption can significantly impact performance metrics, leading to overestimations of efficiency by up to 3.85X in realistic scenarios. The authors identify three key design principles: the importance of channel count in decoding performance, the common near-optimal hierarchy configuration across models, and the limited impact of workload mapping strategies on overall efficiency. These findings underscore the necessity for system-wide co-optimization to achieve substantial efficiency gains in memory-accelerated LLM systems.
Static power consumption can skew efficiency estimates by up to 3.85X, revealing critical oversights in current PIM-GPU design practices for LLM inference.
Heterogeneous DRAM-based processing-in-memory (PIM)-GPU systems promise significant efficiency gains for decode-phase large language model (LLM) inference, particularly in long-output generation, yet current design practices overlook critical factors that determine real-world performance. Through systematic evaluation of diverse architectures and workloads (OPT-7B/70B, Mamba2-2.7B/70B), we reveal three fundamental design principles: (i) static power consumption (DRAM leakage, refresh, and GPU idle power) can dominate the efficiency calculus, causing dynamic-only models to overestimate tokens/s/W by up to 3.85X for realistic deployments (Mamba2-2.7B, batch size 1, 128 input tokens, and 2,048 output tokens); (ii) decoding performance is monotonically non-decreasing with channel count across all evaluated models and workloads, generally plateauing at high channel counts for low-batch workloads; under a fixed-capacity sweep, all models instead share a common near-optimal hierarchy configuration, with substantially larger misconfiguration penalties for attention-based models; (iii) workload mapping strategies provide bounded improvements (up to 14.0%/17.4% kernel-level latency/energy reduction, up to 5.6% end-to-end gain) and are not primary bottlenecks. Significant efficiency gains require system-wide co-optimization. These principles provide design-space guidance for architects designing the next generation of memory-accelerated LLM systems.
