Search papers, labs, and topics across Lattice.
This paper introduces CacheWeaver, a method that optimizes the ordering of evidence in Retrieval-Augmented Generation (RAG) to enhance efficiency and reduce costs associated with prompt length. By maintaining a prefix tree of recently served evidence sequences and employing a greedy algorithm for evidence placement, CacheWeaver achieves a significant reduction in median time-to-first-token (TTFT) by 20-33% across various vLLM configurations. Importantly, this improvement is realized without compromising the quality of answers in QA tests, indicating a strong balance between efficiency and performance.
Efficient evidence ordering can cut response times by up to a third without sacrificing answer quality in RAG systems.
Retrieval-Augmented Generation (RAG) improves factual grounding, but it also lengthens prompts and raises prefill cost. Prefix caching in serving engines such as vLLM reduces this cost only when requests share the same token prefix. In grounded generation, however, adjacent queries may retrieve overlapping evidence in different orders, so set overlap does not become reusable prefix overlap. We present CacheWeaver, a lightweight prompt-layer method for cache-aware evidence ordering. The method keeps a prefix tree over recently served evidence sequences and uses a greedy walk to place the most reusable prefix first, while leaving the serving engine and retrieved evidence set unchanged. Across three vLLM configurations, the method lowers median time-to-first-token (TTFT) by about 20-33 percent relative to retrieval-order prefix caching, without hurting answer quality in our QA tests. The greedy policy reaches 97.5 percent of the median TTFT gain from oracle ordering, indicating that most reusable prefix locality can be recovered by a simple scheduling layer between retrieval and inference.
CMU ML