Search papers, labs, and topics across Lattice.
This paper critiques the reliance on Conceptual Entity Relevance (CER) as a signal for document retrieval, arguing that it often fails to correlate with Observable Entity Relevance (OER), which is more indicative of a document's discriminative power. Through empirical analysis across four datasets, the authors demonstrate that CER and OER have near-zero agreement, highlighting that CER-based supervision tends to select entities that are topically relevant but not effective at distinguishing relevant documents. By aligning retrieval supervision with OER, the study achieves a significant improvement in non-relevant document pruning and overall retrieval performance, suggesting a paradigm shift in entity-aware retrieval methodologies.
Shifting from conceptual to observable relevance can enhance document retrieval effectiveness by up to 10x in non-relevant pruning.
Entity-aware document retrieval uses query-associated entities as ranking signals, assuming that semantically relevant entities are also useful retrieval signals. We show this assumption is insufficient- and explain why. Unlike terms, which are ground-truth observations, entity links are hypotheses produced by an imperfect linker: an entity can be topically central yet provide no discriminative signal if the linker fires indiscriminately across relevant and non-relevant documents. We formalize this as a distinction between Conceptual Entity Relevance (CER)- whether an entity is topically related to a query- and Observable Entity Relevance (OER)- whether its observed presence in a collection discriminates relevant from non-relevant documents. Across four collections and annotation sources including human entity judgments, CER and OER exhibit near-chance agreement ($魏\approx 0$), while OER operationalizations agree substantially ($魏\approx 0.5$), confirming CER as the systematic outlier. CER-based supervision selects topically plausible but weakly discriminative entities, pruning fewer than 4% of non-relevant documents on some collections. Aligning supervision with OER improves non-relevant pruning by up to 10x and open-world MAP by 0.051 over BM25. Our findings motivate a shift from conceptual to observable notions of entity relevance in entity-aware retrieval.
