Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the limitations of Pseudo-Relevance Feedback (PRF) in information retrieval, revealing that while it generally improves retrieval effectiveness, it negatively impacts a significant portion of queries due to query drift. Through a participatory audit involving 108 users, the authors found that only 20.9% of queries benefited from PRF, while 25.6% experienced degraded user experiences, emphasizing the importance of avoiding harm over maximizing gains. The proposed two-stage audit-then-automate framework utilizes LLM-based rerankers to create an auditable system that explains PRF's effects, enhancing transparency and user trust in retrieval processes.
Only 20.9% of queries benefit from Pseudo-Relevance Feedback, highlighting a critical need for explainability in information retrieval systems.
Pseudo-Relevance Feedback (PRF) improves retrieval effectiveness on average, but harms a substantial fraction of queries through query drift, an asymmetry hidden by aggregate offline metrics. Existing Selective PRF (sPRF) approaches typically rely on Query Performance Prediction (QPP) methods derived from the same ranking statistics, and therefore inherit, rather than resolve, this opacity. We argue that this is a core explainability problem in IR, and propose a two-stage audit-then-automate framework. In Stage 1, a participatory audit with 108 users across 43 TREC Deep Learning 2019 queries shows that only 20.9% of queries benefit from PRF, while 25.6% suffer a degraded user experience, and that avoiding harm is nearly twice as valuable as exploiting successful expansion. In Stage 2, we repurpose LLM-based rerankers as system preference predictors that replicate these user-derived labels automatically, grounded in inspectable document evidence. Together, the two stages explain which queries PRF harms, why an sPRF decision is made, and how the decision can be inspected at scale, turning an opaque retrieval component into an auditable, user-grounded one.