Search papers, labs, and topics across Lattice.
This paper introduces a budget-aware evaluation framework for Active Retrieval-Augmented Generation (RAG) systems, emphasizing the importance of utility estimation in determining when to retrieve external knowledge. By analyzing the interplay between retrieval decisions, accuracy, and deployment costs, the authors reveal that traditional evaluations often conflate performance metrics, leading to misleading conclusions about system effectiveness. Key findings indicate that retrieval harm is significant, and simple baselines can perform comparably to more complex learned utility routers, highlighting the need for comprehensive reporting of various evaluation metrics.
Retrieval harm can significantly impact performance, and simple baselines often outperform complex models in Active RAG systems.
Active RAG systems decide when to retrieve external knowledge during generation, making them a budget-sensitive case of agentic RAG and self-adaptive retrieval. Yet evaluations often leave the operating point underspecified: two systems may both claim a 50% evidence-usage budget while realizing different held-out usage rates, so higher accuracy can reflect a looser budget rather than a better retrieval policy. We study budget-aware evaluation for Active RAG by recasting active retrieval as utility estimation, where retrieval is valuable only through its marginal correctness change over a no-retrieval answer. This view separates three questions that single-point evaluations conflate: whether trigger scores rank useful retrieval decisions, whether thresholds calibrated on past data meet future budgets, and how trigger-side computation changes deployment cost. We operationalize these questions with exact top-k utility frontiers, deployable threshold frontiers, conservative budget frontiers, harm audits, and cost decompositions. Across knowledge-intensive multi-hop QA datasets and open instruction models, retrieval harm is non-negligible, router rankings change across datasets and budgets, nominal thresholds can miss target usage, and simple uncertainty or retrieval-score baselines often rival learned utility routers. Budget-aware Active RAG evaluations should therefore report frontiers, realized usage, threshold-transfer error, harm rates, and cost decompositions alongside accuracy.
CMU ML