Search papers, labs, and topics across Lattice.
This paper tackles the memory bottleneck in LLM inference on Processing-in-Memory (PIM) architectures, specifically addressing the large KV cache sizes. They propose AQPIM, a PIM-aware activation quantization framework based on Product Quantization (PQ) optimized for LLMs, performing quantization directly within the PIM memory to leverage its high internal bandwidth. Results show AQPIM achieves a 3.4x speedup over a SOTA PIM approach and drastically reduces GPU-CPU communication, which can account for 90-98.5% of decoding latency.
LLM inference gets a 3.4x speed boost on PIM architectures thanks to a novel quantization method that cuts down on memory bottlenecks by quantizing activations directly in memory.
Processing-in-Memory (PIM) architectures offer a promising solution to the memory bottlenecks in data-intensive machine learning, yet often overlook the growing challenge of activation memory footprint. Conventional PIM approaches struggle with massive KV cache sizes generated in long-context scenarios by Transformer-based models, frequently exceeding PIM's limited memory capacity, while techniques like sparse attention can conflict with PIM's need for data locality. Existing PIM approaches and quantization methods are often insufficient or poorly suited for leveraging the unique characteristics of activations. This work identifies an opportunity for PIMspecialized activation quantization to enhance bandwidth and compute efficiency. We explore clustering-based vector quantization approaches, which align well with activation characteristics and PIM's internal bandwidth capabilities. Building on this, we introduce AQPIM, a novel PIM-aware activation quantization framework based on Product Quantization (PQ), optimizing it for modern Large Language Models (LLMs). By performing quantization directly within memory, AQPIM leverages PIM's high internal bandwidth and enables direct computation on compressed data, significantly reducing both memory footprint and computational overhead for attention computation. AQPIM addresses PQ's accuracy challenges by introducing several algorithmic optimizations. Evaluations demonstrate that AQPIM achieves significant performance improvements, drastically reducing of GPU-CPU communication that can account for $90 \sim 98.5 \%$ of decoding latency, together with $3.4 \times$ speedup over a SOTA PIM approach.
