Search papers, labs, and topics across Lattice.
This paper presents NITRO, a high-performance in-storage computing architecture that enhances NAND flash-based processing-in-memory by integrating a fast DRAM subsystem for intermediate value buffering. This design addresses the latency issues associated with traditional NAND flash memory, which hampers the efficiency of large language models. The proposed architecture achieves up to an 85% reduction in inference latency, significantly improving computational performance in memory-constrained environments.
By buffering intermediate values in fast DRAM, NITRO slashes inference latency by up to 85%, revolutionizing the efficiency of NAND flash-based computing.
In-storage computing (ISC) is considered a next-generation memory architecture for its ability to relieve the data bottleneck between the host and the memory. While the required resources of large language models (LLMs) have increased significantly in recent years, the memory density has not scaled accordingly. Recently, several works have studied NAND flash-based processing-in-memory (NAND-PIM) schemes to exploit the high density of the memory. However, they do not address the dataflow/buffer for the intermediate values, so a simple method is to deal with the values in the slow flash memory array. To overcome such a limitation, we propose a high-performance NAND flash-based ISC architecture with enhanced activation buffering. Instead of using the very slow flash memory array for the intermediate values, our architecture buffers the values in a fast DRAM subsystem. This approach effectively handles the high-latency penalties when activations are programmed into slower TLC NAND flash. We also introduce a distributed dataflow approach for the NAND-PIM array. This approach maximizes computational parallelism by employing efficient intra-plane data mapping. The results show that our proposed architecture achieves significant performance improvements, reducing the inference latency by up to 85% compared to the baseline.
