Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for evaluating and optimizing document sets used by large language models, addressing the limitations of existing relevance-centric retrieval methods that fail to consider inter-document interactions. By developing SetwiseEvalKit, a comprehensive benchmark with 28K evaluation rubrics, the authors systematically assess various reranking methods, revealing that no single approach excels across all dimensions of document set quality. The proposed Rubric4Setwise method effectively translates rubric-based criteria into selection signals, achieving superior downstream generation performance while minimizing the number of documents and search rounds required.
Relying solely on relevance leads to suboptimal document sets; this new framework reveals the critical importance of inter-document relationships in retrieval quality.
As large language models and AI agents become the primary consumers of search results, document set quality determines the upper bound of downstream generation. Yet existing evaluation systems remain confined to scoring documents independently and aggregating via nDCG, ignoring inter-document interactions (redundancy, conflict, complementarity) and unable to answer what makes one document set better than another. To address these issues, we propose a complete evaluate-diagnose-optimize framework. We design SetwiseEvalKit, a three-level, nine-dimension document set evaluation benchmark covering both short-form and long-form scenarios, comprising approximately 28K high-quality evaluation rubrics. We systematically evaluate 12 rerankers: even the best method achieves no more than 45% coverage, cross-document coordination dimensions are universally weak, and no single method maintains top performance across both settings. Building on this, we propose Rubric4Setwise, a training-free method that converts rubric-based evaluation criteria into document set selection signals, achieving the best downstream generation performance with fewer documents and search rounds. It is the only method that maintains state-of-the-art results across both scenarios, validating the effectiveness of closing the loop from evaluation to optimization.
