Search papers, labs, and topics across Lattice.
This paper introduces Rank-Deviation Quality (RDQ), a novel evaluation metric designed for retrieval and ranking systems that accommodates varying numbers of reference items. The metric scores candidate rankings based on their alignment with an ordered reference list, applying a rank-deviation penalty that reflects the importance of both the retrieved items and their order. RDQ outperforms existing metrics, achieving the highest median empirical power on a large point-of-interest dataset and demonstrating strong agreement with full-query orderings, highlighting its effectiveness in evaluating multi-answer retrieval tasks.
RDQ achieves superior evaluation power by effectively balancing the importance of retrieved items and their order, outperforming traditional metrics in multi-answer retrieval scenarios.
We introduce Rank-Deviation Quality (RDQ), an evaluation metric for retrieval and ranking systems that adapts to queries with varying numbers of reference items, from a single correct answer to many valid results. RDQ scores a candidate ranking against an ordered reference list (ORL): each retrieved reference item contributes its output-position weight multiplied by a rank-deviation penalty, and items outside the ORL receive zero credit. Application-specific parameters control tolerance to misordering. Larger values emphasize retrieving valid reference items, whereas smaller values place more weight on matching their reference order. The output-position weights can reflect visibility in the application's interface, such as a vertical list or a carousel. Unlike metrics that require absolute relevance grades, RDQ operates on ordinal rankings, which annotators can produce through pairwise or listwise judgments. Unlike rank-correlation measures such as Kendall's tau, RDQ accounts for both which items are returned and how they are ordered. On a 5,000-query point-of-interest (POI) dataset with 12 systems, RDQ has the highest median empirical power@100 among the 13 evaluated metric configurations. It reaches mean tau>= 0.8 agreement with its own full-query ordering at 200 queries; RBP(0.9), the strongest tested non-RDQ configuration, reaches the same threshold at 250. On TREC Deep Learning benchmarks, where NDCG uses native graded labels and RDQ uses ordinal tiers derived from them, RDQ reaches comparable median power at n=25, while NDCG is higher at n=100.
Amazon Science