Search papers, labs, and topics across Lattice.
This paper introduces Confidence-Aware Reranking (CAR), a training-free RAG reranking framework that prioritizes documents based on their ability to increase the generator's confidence, as measured by the semantic consistency of sampled answers. CAR promotes documents that increase confidence, demotes those that decrease it, and uses a query-level gate to avoid unnecessary interventions. Experiments on BEIR datasets demonstrate that CAR consistently improves NDCG@5 across various retrievers, rerankers, and LLM backbones, with ranking gains strongly correlating with downstream generation F1 improvements.
RAG systems can be significantly improved by reranking documents based on how much they increase the LLM's confidence, not just their relevance.
Retrieval-Augmented Generation (RAG) depends on document ranking to provide useful evidence for generation, but conventional reranking methods mainly optimize query-document relevance rather than generation usefulness. A relevant document may still introduce noise, while a lower-ranked document may better reduce the generator's uncertainty. We propose CAR (Confidence-Aware Reranking), a query-guided, training-free, and plug-and-play reranking framework that uses generator confidence change as a document usefulness signal. CAR estimates confidence through the semantic consistency of multiple sampled answers under query-only and query-document conditions. Documents that significantly increase confidence are promoted, those that decrease confidence are demoted, and uncertain cases preserve the baseline order, while a query-level gate avoids unnecessary intervention on already confident queries. Experiments on four BEIR datasets show that CAR consistently improves NDCG@5 across sparse and dense retrievers, LLM-based and supervised rerankers, and four LLM backbones. Notably, CAR improves the YesNo reranker by 25.4 percent on average under Contriever retrieval, and its ranking gains strongly correlate with downstream generation F1 improvements, achieving Spearman rho = 0.964.
