Search papers, labs, and topics across Lattice.
This paper introduces CoinRAG, a novel approach that enhances Retrieval-Augmented Generation (RAG) by optimizing the reuse of fine-grained contextual information from KV caches, rather than relying on coarse-grained chunk encoding. By implementing a two-stage retrieval process, CoinRAG identifies and assembles relevant semantic units, leading to a more efficient and semantically rich contextual representation. The method achieves a 5.3% relative improvement in answer quality while significantly reducing operational costs, establishing a new Pareto frontier under low latency constraints.
CoinRAG redefines efficiency in RAG by achieving higher accuracy with lower operational costs through innovative cache reuse strategies.
Recent optimization studies on Retrieval-Augmented Generation (RAG) have exploited chunk-level KV cache reuse to avoid processing long retrieved contexts for higher efficiency, while significant information redundancy and noise still remain in the coarse-grained chunks. This paper optimizes the Pareto frontier under low prefill latency constraints while maximizing accuracy by proposing CoinRAG (Contextualized Information Nugget KV Cache Reuse for Long-Context RAG). The name metaphorically reflects our core mechanism: much like assembling small tokens (or"coins") to accumulate a larger value, CoinRAG compositionally reuses offline-computed, fine-grained nugget caches to form a learned contextual representation efficiently in a more semantically relevant but compact manner. Specifically, instead of full-chunk encoding, CoinRAG identifies query-relevant semantic units within retrieved chunks through two-stage retrieval and seamlessly assembles their sliced KV representations with a chunk-level context. Extensive evaluations on LongBench multi-hop question answering tasks demonstrate that CoinRAG significantly reduces operational costs and outperforms the other baselines with a new Pareto frontier and an average 5.3% relative improvement in answer quality (F1) under a standard fast prefill latency budget.
