Search papers, labs, and topics across Lattice.
This study critically evaluates the effectiveness of anchor-based pointwise LLM rerankers, specifically the GCCP/PAGC method, through a reproduction-first approach that uncovers undocumented implementation details affecting performance. The authors achieve a significant improvement in nDCG@10 from 0.24 to 0.66 by addressing these details and conduct a controlled analysis revealing that while contrastive scoring is robust, certain design choices are less effective than previously thought. Notably, the combination of contrastive and standard relevance scores benefits weaker retrievers like BM25 but offers minimal advantage with stronger models, and a simpler anchor construction method proves sufficient or superior to the original complex approach.
Contrastive scoring is key to the success of anchor-based reranking, but the complexity of anchor design may be overhyped, especially with stronger retrievers.
Anchor-based pointwise LLM reranking scores each candidate against a shared reference passage to recover cross-document context at pointwise cost. We study when this actually helps, using GCCP/PAGC as a representative method. Our study is reproduction-first. We use reproduction as a starting point for a controlled component-level stress test of anchor-based pointwise reranking. Our initial reimplementation, based only on the paper text, achieves 0.24 nDCG@10 instead of the reported 0.66, revealing that several undocumented implementation details are necessary to reproduce the method. After identifying and recovering eight such details, we reproduce the reported results within 1.6% and use the validated implementation for controlled analysis. We find that the core contrastive scoring idea is robust under rigorous statistical correction. However, two design choices held fixed in the original paper are less reliable. First, we find that combining the contrastive score with the standard pointwise relevance score helps when the first-stage retriever is BM25, but gives little or no benefit when the first-stage retriever is a stronger dense model such as E5. Second, the paper's more complex method for constructing the anchor is unnecessary. A much simpler anchor, built by interleaving the top-ranked sentences, matches or outperforms it across datasets. These findings are consistent across different LLM backbones, including a 4-bit quantized 72B model. Overall, anchor-based pointwise reranking is effective, but its gains come mainly from contrastive scoring rather than from the more complex aggregation and anchor-construction choices, and they appear under narrower conditions than the original evaluation suggests.
