Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
EdgeCIM is a hardware-software co-design framework featuring a 65nm compute-in-memory (CIM) macro and tile-based mapping strategy optimized for accelerating the autoregressive decoding phase of small language models (SLMs) on edge devices. By balancing pipeline stages and maximizing parallelism, EdgeCIM alleviates DRAM bandwidth bottlenecks inherent in GEMV operations. Benchmarks across various SLMs (up to 4B parameters) demonstrate that EdgeCIM achieves significant throughput and energy efficiency gains compared to NVIDIA Orin Nano and Qualcomm SA8255P, reaching an average of 336.42 tokens/s and 173.02 tokens/J under INT4 precision.
Forget GPUs, this CIM-based accelerator delivers up to 49x better energy efficiency for running LLaMA3.2 on edge devices.
The growing demand for deploying Small Language Models (SLMs) on edge devices, including laptops, smartphones, and embedded platforms, has exposed fundamental inefficiencies in existing accelerators. While GPUs handle prefill workloads efficiently, the autoregressive decoding phase is dominated by GEMV operations that are inherently memory-bound, resulting in poor utilization and prohibitive energy costs at the edge. In this work, we present EdgeCIM, a hardware-software co-design framework that rethinks accelerator design for end-to-end decoder-only inference. At its core is a CIM macro, implemented in 65nm, coupled with a tile-based mapping strategy that balances pipeline stages, maximizing parallelism while alleviating DRAM bandwidth bottlenecks. Our simulator enables design space exploration of SLMs up to 4B parameters, identifying Pareto-optimal configurations in terms of latency and energy. Compared to an NVIDIA Orin Nano, EdgeCIM achieves up to 7.3x higher throughput and 49.59x better energy efficiency on LLaMA3.2-1B, and delivers 9.95x higher throughput than Qualcomm SA8255P on LLaMA3.2-3B. Extensive benchmarks on TinyLLaMA-1.1B, LLaMA3.2 (1B, 3B), Phi-3.5-mini-3.8B, Qwen2.5 (0.5B, 1.5B, 3B), SmolLM2-1.7B, SmolLM3-3B, and Qwen3 (0.6B, 1.7B, 4B) reveal that our accelerator, under INT4 precision, achieves on average 336.42 tokens/s and 173.02 tokens/J. These results establish EdgeCIM as a compelling solution towards real-time, energy-efficient edge-scale SLM inference.