Search papers, labs, and topics across Lattice.
This paper critiques the conventional assumption that reranking retrieved examples in few-shot selection always enhances performance, revealing that it can actually degrade outcomes. The authors introduce Training-Free Gated Reranking, a method that utilizes model uncertainty to determine whether reranking is beneficial. Their extensive experiments across multiple LLMs and datasets show that this approach not only reduces computational costs by 15%-80% but also improves average performance by up to 2%, highlighting the importance of targeted reranking based on uncertainty levels.
Reranking can hurt performance, but targeting high-uncertainty instances can yield significant gains while slashing computational costs.
Few-shot selection typically assumes that reranking retrieved examples always improves performance. We challenge this view by identifying that the expensive reranking step can in fact degrade performance. Instead, we propose \emph{Training-Free Gated Reranking}, which decides whether to rerank the few-shot examples based on the model's uncertainty. Extensive experiments across 8 LLMs, covering 7 NLU datasets and 9 MT domain-language combinations, demonstrate that our approach reduces computational costs by 15\%-80\% while improving average performance by up to 2\%. These findings indicate that higher computational cost does not guarantee better performance, and that reranking is most beneficial when targeted at high-uncertainty instances.
