Search papers, labs, and topics across Lattice.
This paper introduces PRQ-KMeans, a novel approach to semantic identifier tokenization that enhances generative retrieval and recommendation by addressing limitations in existing residual-quantization tokenizers. By removing the global-mean component and employing Top-k similarity-weighted centroid updates, PRQ-KMeans effectively captures hierarchical token sequences while improving performance metrics such as HitRate and MRR. Experiments demonstrate that this method outperforms traditional tokenizers, achieving up to 7.4% improvement in HitRate and 11.8% in MRR on a large-scale industrial search dataset.
PRQ-KMeans achieves up to 11.8% improvement in recommendation performance by rethinking how semantic identifiers are tokenized.
Semantic identifiers (SIDs) represent entities as hierarchical token sequences for generative retrieval and recommendation. Residual-quantization tokenizers construct these sequences by selecting a codeword at each level and passing a residual to the next. We view this process as progressive commonality removal: each token captures a component shared within its group, while later tokens should model the remaining differences. This view reveals three limitations: a corpus-wide shared component can consume first-level capacity, hard assignment ignores graded similarities to nearby codewords, and full-codeword subtraction can leave variation along the selected-codeword direction in the next residual. We therefore develop our solution in the post-hoc setting, where residual construction is not constrained by input reconstruction. Specifically, we propose PRQ-KMeans, which removes the global-mean component, refines centroids with Top-k similarity-weighted updates, and replaces full-codeword subtraction with a projection residual that removes each representation's selected-centroid component. Experiments on a large-scale industrial search dataset and four public recommendation benchmarks show that PRQ-KMeans achieves the strongest overall performance among the evaluated tokenizers, including gains of up to 7.4% in HitRate and 11.8% in MRR on the industrial dataset.
