Search papers, labs, and topics across Lattice.
This paper introduces SIDInspector, a diagnostic tool designed to evaluate Semantic-ID tokenizers by providing a mapping-first approach to inspect item-to-code mappings and their associated metadata. The tool identifies critical issues such as coverage gaps and aliasing before downstream training, enabling researchers to optimize their tokenizers for better performance. Key findings reveal that while some mappings exhibit high aliasing rates, deterministic prefix controls outperform learned exports in terms of alignment, suggesting the need for separate inspections of addressability and meaningful prefixes.
High aliasing rates in Semantic-ID tokenizers can be detected before training, revealing that deterministic prefix controls significantly outperform learned mappings in alignment.
Semantic-ID (\sid) tokenizers are increasingly reused as standalone artifacts in generative recommendation: an exported item-to-code mapping becomes the address space that a later sequence generator must use. These mappings rarely come with a common inspection interface, so coverage gaps, full-code aliasing, behaviorally weak prefixes, tail compression, and prefix fan-out are often found only after downstream training. We present \tool, a mapping-first diagnostic resource for \sid tokenizer artifacts. \tool defines a small adapter contract over item mappings, metadata, interactions, and optional generator traces; validates the contract; and reports mapping-level probes for utilization, aliasing, neighborhood alignment, popularity allocation, and structural cost, with hooks for temporal churn and generator traces. \tool reports inspectable artifact profiles before downstream leaderboard scores. The released resource covers four tokenizer artifact lines: a same-item GRID/RQ-KMeans-style and ReSID/GAOQ contrast on 23,742 Musical items, plus released LETTER and LC-Rec item-index artifacts. In the Musical contrast, the GRID-style feature-text export has 3,749 unique full codes and a 0.977 full-code aliasing rate, while ReSID/GAOQ is aliasing-free in its exported mapping. Yet the strongest prefix--co-occurrence alignment comes from a deterministic category-prefix control, not from either learned export row (0.447 versus 0.154 and 0.055--0.080), showing that addressability and behaviorally meaningful prefixes should be inspected separately. Cross-domain, fixed-reranker, and mechanism-probe checks support the same diagnostic direction: prefix alignment is a candidate-exposure signal, while final ranking quality remains a downstream model question.
