Search papers, labs, and topics across Lattice.
This paper introduces Rank-Deviation Quality (RDQ), a novel evaluation metric designed for multi-answer retrieval and ranking systems that accommodates varying numbers of reference items. RDQ scores candidate rankings based on their alignment with an ordered reference list, applying a rank-deviation penalty that adjusts according to application-specific parameters, thus enabling a nuanced assessment of retrieval effectiveness. The results demonstrate that RDQ outperforms existing metrics in empirical power and agreement with full-query orderings, particularly in diverse datasets and benchmarks.
RDQ reveals that a distance-aware metric can significantly enhance the evaluation of retrieval systems by prioritizing both the relevance and the order of retrieved items.
We introduce Rank-Deviation Quality (RDQ), an evaluation metric for retrieval and ranking systems that adapts to queries with varying numbers of reference items, from a single correct answer to many valid results. RDQ scores a candidate ranking against an ordered reference list (ORL): each retrieved reference item contributes its output-position weight multiplied by a rank-deviation penalty, and items outside the ORL receive zero credit. Application-specific parameters control tolerance to misordering. Larger values emphasize retrieving valid reference items, whereas smaller values place more weight on matching their reference order. The output-position weights can reflect visibility in the application's interface, such as a vertical list or a carousel. Unlike metrics that require absolute relevance grades, RDQ operates on ordinal rankings, which annotators can produce through pairwise or listwise judgments. Unlike rank-correlation measures such as Kendall's tau, RDQ accounts for both which items are returned and how they are ordered. On a 5,000-query point-of-interest (POI) dataset with 12 systems, RDQ has the highest median empirical power@100 among the 13 evaluated metric configurations. It reaches mean tau>= 0.8 agreement with its own full-query ordering at 200 queries; RBP(0.9), the strongest tested non-RDQ configuration, reaches the same threshold at 250. On TREC Deep Learning benchmarks, where NDCG uses native graded labels and RDQ uses ordinal tiers derived from them, RDQ reaches comparable median power at n=25, while NDCG is higher at n=100.
Amazon Science