Search papers, labs, and topics across Lattice.
This paper investigates the performance of generative retrieval models on the LIMIT synthetic dataset, which was designed to expose theoretical limitations of dense retrieval. Generative retrieval models (SEAL and MINDER) significantly outperform dense retrieval and even BM25 on the original LIMIT dataset. However, the study reveals that generative retrieval struggles with identifier ambiguity when presented with hard negative samples in an extended LIMIT dataset, highlighting a vulnerability in the decoding mechanism.
Generative retrieval models, while promising, can be easily tripped up by identifier ambiguity, causing a significant performance drop when faced with simple hard negatives.
While dense retrieval models, which embed queries and documents into a shared low-dimensional space, have gained widespread popu- larity, they were shown to exhibit important theoretical limitations and considerably lag behind traditional sparse retrieval models in certain settings. Generative retrieval has emerged as an alternative approach to dense retrieval by using a language model to predict query-document relevance directly. In this paper, we demonstrate strengths and weaknesses of generative retrieval approaches us- ing a simple synthetic dataset, called LIMIT, that was previously introduced to empirically demonstrate the theoretical limitations of embedding-based retrieval but was not used to evaluate genera- tive retrieval. We close this research gap and show that generative retrieval achieves the best performance on this dataset without any additional training required (0.92 and 0.99 R@2 for SEAL and MINDER, respectively), compared to dense approaches (<0.03 Re- call@2) and BM25 (0.86 R@2). However, we then proceed to extend the original LIMIT dataset by adding simple hard negative samples and observe the performance degrading for all the models including the generative retrieval models (0.51 R@2) as well as BM25 (0.21 R@2). Error analysis identifies a failure in the decoding mechanism, caused by the inability to produce identifiers that are unique to relevant documents. Future generative retrieval must address these issues, either by designing identifiers that are more suitable to the decoding process or by adapting decoding and scoring algorithms to preserve relevance signals.
