Search papers, labs, and topics across Lattice.
This paper introduces CHIPSMORE, an innovative accelerator designed for large language model (LLM) inference that integrates compute-in-interconnect and compute-in-memory technologies to enhance performance across various workloads. By utilizing a heterogeneous architecture with resistive RAM and static RAM, along with a dynamic memory allocation scheme, CHIPSMORE achieves significant improvements in throughput and energy efficiency, outperforming existing solutions like the Nvidia H100. The evaluation shows that CHIPSMORE can deliver up to 2.38 times higher throughput and 27 times greater energy efficiency for Mistral-7B inference, all while eliminating the need for weight replication in multi-request scenarios.
CHIPSMORE accelerates LLM inference by achieving 2.38x higher throughput and 27x better energy efficiency compared to Nvidia H100, without duplicating weights for concurrent requests.
Large language model (LLM) inference exhibits substantial variability across adaptation modes, context lengths, and request concurrency, creating challenges for maintaining high utilization, memory efficiency, and scalable performance on compute-in-memory (CIM) accelerators. This paper presents CHIPSMORE, a multi-mode and multi-request LLM inference accelerator that integrates compute-in-interconnect and CIM to support both base-mode and low-rank adaptation (LoRA) inference under diverse workloads. CHIPSMORE employs heterogeneous processing elements consisting of resistive RAM analog compute-in-memory (RRAM-ACIM) and static RAM digital compute-in-memory (SRAM-DCIM) interconnected through a programmable Inter-PE computational network (IPCN). A composable hierarchical key-value (KV) memory scheme dynamically allocates router scratchpad, SRAM-DCIM, and embedded DRAM (eDRAM) resources according to workload requirements, enabling scalable support for long-context and batched inference. Furthermore, a non-replicated multi-request execution pipeline exploits request-level parallelism without duplicating pretrained weights, while a state-aware resource reconfiguration mechanism selectively retains runtime states and power-gates inactive resources to improve energy efficiency. Evaluation using cycle-accurate hardware-software co-simulation demonstrates that CHIPSMORE effectively sustains high throughput across varying model sizes, context lengths, and batch sizes while maintaining favorable power scaling. Compared with Nvidia H100, CHIPSMORE achieves up to $2.38\times$ higher throughput and $27\times$ higher energy efficiency on Mistral-7B inference while eliminating weight replication for multi-request serving.
