Search papers, labs, and topics across Lattice.
This paper evaluates various Retrieval-Augmented Generation (RAG) frameworks, including regular RAG, GraphRAG, Modular RAG, and Agentic RAG, through a comprehensive set of nine standardized scenarios tailored for semi-structured knowledge bases. The authors introduce a novel context engineering method that significantly reduces token usage by 19%-53% while addressing context overflow issues, thereby enhancing the efficiency of text and graph retrievals. Importantly, the findings reveal a retrieval-generation gap, indicating that increased retrieval capabilities do not necessarily translate to improved generation quality, which has implications for the design of intelligent RAG systems.
Advanced RAG methods like GraphRAG and Agentic RAG can reduce token usage by up to 53%, but they don't always enhance generation quality as expected.
As advanced RAG variants like GraphRAG and Agentic RAG emerge, one leading question is when and how to use them. Here, we introduce a framework for different RAG scenarios evaluation and comparison on semi-structured knowledge bases, including regular RAG, GraphRAG, Modular RAG and Agentic RAG. We provide implementation for 9 standardized RAG scenarios, and conduct experiments for a comprehensive comparison. These scenarios are designed for real use cases regarding data and domain restrictions, spanning from simple document-based retrieval to advanced features such as hybrid text-graph retrieval, integration with computed or pre-defined domain knowledge graphs, agentic multi-step planning, and agent-graph integration. Besides, we present a novel context engineering method for GraphRAG and Agentic RAG, addressing the context/memory overflow issues, efficiently managing text and graph retrievals with new representations and agentic loop design, leading to 19%-53% reduction on token usage. Moreover, further analysis identifies a retrieval-generation gap where expanded retrieval does not proportionally improve generation quality, suggesting retrieval-oriented metrics overstate advanced retrieval benefits. This work provides data-driven insights on when and how to use them for building production-ready intelligent RAG systems.
Amazon Science
Tsinghua AI