Search papers, labs, and topics across Lattice.
This paper investigates the inconsistent results in entity-oriented retrieval by analyzing the impact of evaluation settings and entity linking quality. They find that current evaluation protocols conflate conditional (entity-restricted) and open-world settings, leading to misleading conclusions about model performance. Their analysis reveals that the primary bottleneck is the limited coverage and discriminative power of entity signals, rather than the neural reranker architecture itself.
Entity-oriented retrieval's inconsistent performance isn't a model problem, but a data problem: even perfect entity selection only covers 19.7% of relevant documents, and current supervision strategies optimize for semantic relevance at the expense of corpus-grounded discriminativeness.
Entity-oriented retrieval assumes that relevant documents exhibit query-relevant entities, yet evaluations report conflicting results. We show this inconsistency stems not from model failure, but from evaluation. On TREC Robust04, we evaluate six neural rerankers and 437 unsupervised configurations against BM25. Across 443 systems, none improves MAP by more than 0.05 under open-world evaluation over the full candidate set, despite strong gains under entity-restricted settings. The best configuration matches the official Robust04 best system and outperforms most neural rerankers, indicating that the architecture is not the limiting factor. Instead, the bottleneck is the entity channel: even under idealized selection, entity signals cover only 19.7\% of relevant documents, and no method achieves both high coverage and discrimination. We explain this via a distinction between Conceptual Entity Relevance (CER) -- semantic relatedness -- and Observable Entity Relevance (OER) -- corpus-grounded discriminativeness under a given linker. All supervision strategies operate at the CER level and ignore the linking environment, leading to signals that are semantically valid but not discriminative. Improving supervision therefore does not recover open-world performance: stronger signals reduce coverage without improving effectiveness. Conditional and open-world evaluation answer different questions: exploiting entity evidence versus improving retrieval under realistic linking, but are often conflated. Progress requires datasets with entity-level discriminativeness and evaluation that reports both coverage and effectiveness. Until then, conditional gains do not imply open-world effectiveness, and open-world failures do not invalidate entity-based models.
