Search papers, labs, and topics across Lattice.
This paper investigates the reliability of Semantic-ID (SID) tokenizers in generative recommendation, revealing a critical flaw where multiple distinct items are mapped to the same SID sequence due to tokenizer compression. This "collision" problem leads to inflated Hit@K metrics, overestimating the true item-level recommendation performance by up to 103%. To address this, the authors introduce collision-aware item-level metrics and a post-tokenizer reassignment procedure to create collision-free SID assignments, enabling more accurate evaluation.
Semantic-ID tokenizers, a promising approach for generative recommendation, can inflate Hit@K metrics by over 100% due to collisions where multiple items map to the same token sequence.
In Semantic-ID (SID) based generative recommendation, each item is represented as a sequence of discrete codes, and an autoregressive model is trained to generate the SID sequence of the next item; top-K performance is then measured by checking whether the SID sequence of the target item appears among the generated sequences. This evaluation protocol equates SID-level matching with item-level recommendation, an equivalence that holds only when every SID sequence maps to a single item. We show this assumption breaks down in practice: because tokenizers compress item features into a code space, semantically similar but collaboratively distinct items are frequently assigned the same SID sequence. Across four datasets and five representative tokenizers, the fraction of items involved in such collisions reaches 30.5%, so matching a shared SID sequence identifies only a collision group rather than the target item. Consequently, SID-level metrics overestimate item-level performance (Hit@10 is inflated by up to 103.36%), and the inflation grows with the collision rate. To support faithful comparison, we develop collision-aware item-level metrics computed directly from generated SID sequences, together with a post-tokenizer procedure that reassigns last-level SIDs at minimum cost to obtain a collision-free assignment for any existing tokenizer. Our results indicate that SID-level rankings in prior work should be interpreted with caution, and that reliable tokenizer evaluation requires either item-level correction or collision-free SID assignments.
