Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the unique failure mechanisms of Generative Retrieval (GR) systems, particularly focusing on n-gram-based methods like SEAL and MINDER. By developing a comprehensive taxonomy of GR failure modes and conducting empirical analyses, the authors identify critical issues such as ambiguous document identifiers and low diversity in generated identifiers. Additionally, they introduce a web-based tool designed to aid researchers in diagnosing and understanding the failures of n-gram generation in retrieval tasks.
N-gram-based generative retrieval systems suffer from ambiguous identifiers and low diversity, impacting their effectiveness in document retrieval.
Generative Retrieval (GR) is an emerging Information Retrieval (IR) paradigm that is motivated by increasingly capable language models. In GR, a model directly generates identifiers for relevant documents. While these systems offer unique advantages, they also introduce distinct failure mechanisms. We explore these failure modes in three contributions: (1) We present a taxonomy of GR failure modes based on GR literature. (2) We empirically investigate failure in a subset of GR: ngram-based methods, more specifically, SEAL and MINDER. Our analysis reveals common issues, such as ambiguous docids, low identifier diversity, and the disproportionate impact of specific identifiers. (3) We introduce a new web-based tool that helps the IR community analyze generated ngrams and their respective contribution to the final ranking, providing an intuitive interface to identify where such GR methods go wrong.