Search papers, labs, and topics across Lattice.
This paper introduces variable-length semantic identifiers (VLSIDs) for recommender systems to address the limitations of fixed-length semantic IDs, which are inefficient and misaligned with the skewed frequency structure of real-world catalogs. The authors draw inspiration from emergent communication to develop a discrete variational autoencoder with Gumbel-Softmax reparameterization that learns item representations of adaptive length. Experiments demonstrate that VLSIDs offer a more efficient and principled approach to representing items in recommender systems compared to fixed-length semantic IDs.
Stop wasting tokens: Variable-length semantic IDs dramatically improve recommender system performance by adapting item representation length to item popularity, mirroring how humans communicate.
Generative models are increasingly used in recommender systems, both for modeling user behavior as event sequences and for integrating large language models into recommendation pipelines. A key challenge in this setting is the extremely large cardinality of item spaces, which makes training generative models difficult and introduces a vocabulary gap between natural language and item identifiers. Semantic identifiers (semantic IDs), which represent items as sequences of low-cardinality tokens, have recently emerged as an effective solution to this problem. However, existing approaches generate semantic identifiers of fixed length, assigning the same description length to all items. This is inefficient, misaligned with natural language, and ignores the highly skewed frequency structure of real-world catalogs, where popular items and rare long-tail items exhibit fundamentally different information requirements. In parallel, the emergent communication literature studies how agents develop discrete communication protocols, often producing variable-length messages in which frequent concepts receive shorter descriptions. Despite the conceptual similarity, these ideas have not been systematically adopted in recommender systems. In this work, we bridge recommender systems and emergent communication by introducing variable-length semantic identifiers for recommendation. We propose a discrete variational autoencoder with Gumbel-Softmax reparameterization that learns item representations of adaptive length under a principled probabilistic framework, avoiding the instability of REINFORCE-based training and the fixed-length constraints of prior semantic ID methods.
