Search papers, labs, and topics across Lattice.
The paper investigates whether LLM-based relevance judgment systems (LLM-RJS) can outperform neural embedding retrieval systems (NERS) in information retrieval by leveraging reasoning to overcome the limitations of similarity-based relevance estimation. Using the TREC-DL 2019 dataset, the authors compare LLM-RJS with and without reasoning against NERS, finding no initial improvement. However, further analysis reveals that human annotations also exhibit "short-sightedness," leading to false positives in reasoning-based LLM-RJS, suggesting the potential of LLM-RJS cannot be accurately evaluated with current datasets.
LLMs may secretly be better at information retrieval than embedding similarity suggests, but current datasets are too "short-sighted" to prove it.
With the emergence of Large Language Models (LLMs), new methods in Information Retrieval are available in which relevance is estimated directly through language understanding and reasoning, instead of embedding similarity. We argue that similarity is a short-sighted interpretation of relevance, and that LLM-Based Relevance Judgment Systems (LLM-RJS) (with reasoning) have potential to outperform Neural Embedding Retrieval Systems (NERS) by overcoming this limitation. Using the TREC-DL 2019 passage retrieval dataset, we compare various LLM-RJS with NERS, but observe no noticeable improvement. Subsequently, we analyze the impact of reasoning by comparing LLM-RJS with and without reasoning. We find that human annotations also suffer from short-sightedness, and that false-positives in the reasoning LLM-RJS are primarily mistakes in annotations due to short-sightedness. We conclude that LLM-RJS do have the ability to address the short-sightedness limitation in NERS, but that this cannot be evaluated with standard annotated relevance datasets.
