Search papers, labs, and topics across Lattice.
This paper introduces the Cluster-Ranked Identifier (CRID) method, which improves document retrieval in Generative Retrieval (GR) by decoupling document identifiers into semantic clusters and business-value rankings. By addressing the collision issues of traditional discrete representation learning, CRID enables collision-free identifiers that can be incrementally updated through intra-cluster reranking. Experimental results on a large e-commerce dataset demonstrate that CRID outperforms existing embedding-based retrieval methods, achieving a +1.06% increase in Gross Merchandise Value (GMV) in real-world applications.
Collision-free document identifiers that boost retrieval performance and business value could redefine how we approach generative retrieval systems.
Generative Retrieval (GR) formulates retrieval as a sequence-to-sequence generation task, assigning each document a document identifier (DocID) and retrieving it through autoregressive decoding, making DocID design a critical factor in retrieval quality. However, existing schemes based on discrete representation learning suffer from inherent collision issues and create a mismatch between the DocID's encoding objective and the system's business optimization target. To address these limitations, we propose Cluster-Ranked Identifier (CRID), which decouples DocID into semantic clustering and business-value ranking, yielding collision-free identifiers that support incremental updates via intra-cluster reranking. We further introduce an analytical framework that decomposes retrieval gains into personalized preference and statistical prior generalization, revealing how semantic cluster size governs the balance between the two components. Experiments on a 300M-item Taobao e-commerce corpus show that CRID surpasses the strongest embedding-based retrieval baseline on top-K Hitrate, and delivers +1.06% GMV in full-traffic deployment.
