Search papers, labs, and topics across Lattice.
This study introduces BeyondUncertainty, a method that leverages verbalized confidence from language models to optimize retrieval-augmented generation for knowledge-intensive question answering. By selectively routing retrieval based on confidence estimates, the approach reduces unnecessary computation and irrelevant evidence, achieving a mean token-level F1 score of 0.483 compared to 0.467 for indiscriminate retrieval methods. The findings indicate that while the method enhances performance and reduces passage retrieval by over 20%, it does incur a trade-off in token efficiency due to increased total token usage.
Confidence-based retrieval routing can boost QA performance while cutting down on irrelevant evidence and computation.
Retrieval-augmented generation improves knowledge-intensive question answering, but indiscriminate retrieval can introduce irrelevant evidence and unnecessary computation. We investigate whether verbalized confidence from black-box language models can serve as an actionable signal for retrieval routing. Our method, BeyondUncertainty, first elicits a structured provisional answer and confidence estimate, then applies a model-specific threshold selected on held-out validation data and frozen before test evaluation. Low-confidence questions receive top-5 TF-IDF retrieval followed by a second answer call, whereas high-confidence questions return the provisional answer directly. We evaluate 27,000 policy instances across six QA benchmarks, three model families, and three retrieval policies. BeyondUncertainty achieves 0.483 mean token-level F1, compared with 0.467 for always retrieval and 0.401 for no retrieval, while reducing retrieved passages by 20.4\% relative to always retrieval. When matched on the number of questions routed to retrieval within each dataset-model cell, it outperforms a post-hoc random allocation in 17 of 18 settings, with an average gain of 0.024 F1. Although poorly calibrated as an absolute probability, probe uncertainty modestly predicts question-level retrieval benefit (AUROC = 0.628). However, the additional probe increases total token usage by 28.2\%, revealing a trade-off between more selective evidence acquisition and end-to-end token efficiency.
