Search papers, labs, and topics across Lattice.
The paper introduces CQ-CiM, a hardware-aware data shaping framework that jointly optimizes compression and quantization of embeddings for Compute-in-Memory (CiM) architectures used in Retrieval-Augmented Generation (RAG). CQ-CiM addresses the "representation gap" between high-precision embeddings and the constraints of low-precision, low-dimension CiM arrays. By jointly considering compression and quantization, CQ-CiM improves data fidelity and enables efficient deployment of RAG on edge devices using diverse CiM implementations.
Finally, a way to make Retrieval-Augmented Generation (RAG) on edge devices practical: CQ-CiM jointly compresses and quantizes embeddings to fit the constraints of diverse Compute-in-Memory (CiM) architectures.
Deploying Retrieval-Augmented Generation (RAG) on edge devices is in high demand, but is hindered by the latency of massive data movement and computation on traditional architectures. Compute-in-Memory (CiM) architectures address this bottleneck by performing vector search directly within their crossbar structure. However, CiM's adoption for RAG is limited by a fundamental ``representation gap,'' as high-precision, high-dimension embeddings are incompatible with CiM's low-precision, low-dimension array constraints. This gap is compounded by the diversity of CiM implementations (e.g., SRAM, ReRAM, FeFET), each with unique designs (e.g., 2-bit cells, 512x512 arrays). Consequently, RAG data must be naively reshaped to fit each target implementation. Current data shaping methods handle dimension and precision disjointly, which degrades data fidelity. This not only negates the advantages of CiM for RAG but also confuses hardware designers, making it unclear if a failure is due to the circuit design or the degraded input data. As a result, CiM adoption remains limited. In this paper, we introduce CQ-CiM, a unified, hardware-aware data shaping framework that jointly learns Compression and Quantization to produce CiM-compatible low-bit embeddings for diverse CiM designs. To the best of our knowledge, this is the first work to shape data for comprehensive CiM usage on RAG.
