Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the discrepancy between the perceived need for LLM-augmented retrieval based on synthetic queries and the actual requirements in a production RAG system. Through an analysis of 20,000 query-workflow pairs from the Danish National Encyclopedia, the authors identify a "Coverage Illusion" where synthetic queries overestimate the necessity of LLM augmentation. They then propose a post-retrieval cascade that dynamically escalates to LLM augmentation only when simpler methods fail, achieving improved quality, reduced latency, and significantly lower LLM usage compared to always-on augmentation.
You're wasting LLM inference on most RAG queries: a simple post-retrieval cascade can slash LLM usage by 72% while *improving* quality and latency.
In modern RAG pipelines, query augmentation methods such as HyDE and query expansion are applied to every query, resulting in substantial LLM inference costs and increased end-to-end latency. The empirical justification for this overhead in real production traffic remains largely unexplored. We present a case study of the Danish National Encyclopedia, evaluating five retrieval workflows over 20,000 query-workflow pairs from production traffic and synthetic conditions. In this system, synthetic queries suggest that LLM augmentation is needed for over 90% of queries to achieve high retrieval coverage. However, under our production deferral policy, only 27.8% of real user queries need LLM augmentation. We call this gap the Coverage Illusion and attribute it to a structural mismatch between synthetic and real query distributions. Pre-retrieval routing cannot resolve this gap, as the need for LLM augmentation is only revealed after searching the index, a result confirmed by our evaluation of four machine learning paradigms. The coverage gap, undetectable from the query alone, motivates a post-retrieval cascade that runs workflows in cheapest-first order and escalates to LLM augmentation only when a step returns no documents. Operating entirely without training overhead or secondary serving infrastructure, the cascade improves quality by +0.140 Composite Overall points over Always-HyDE, reduces latency by 31.8%, and serves 72.2% of real user queries without LLM augmentation.