Search papers, labs, and topics across Lattice.
This paper introduces Semantic Subword Tokenization (SST) for generative recommender systems, which addresses the issue of Intra-item Attention Overload by representing historical items as variable-length semantic subwords. By merging adjacent atom tokens into compact subwords and incorporating behavior-induced co-occurrence signals, SST shifts the focus from low-level dependencies to high-level inter-item transitions. Experimental results across multiple datasets demonstrate that SST significantly outperforms traditional fixed-length and variable-length tokenization methods in next-item prediction tasks.
Shifting from fixed-length tokens to semantic subwords can dramatically enhance the efficiency of attention mechanisms in generative recommenders.
In generative recommender systems, items are typically tokenized into fixed-length semantic ID sequences for autoregressive next-item prediction. However, for user-context modeling, this fine-grained representation triggers Intra-item Attention Overload: excessive attention is spent on low-level intra-item dependencies rather than high-level inter-item behavioral transitions. To address this, we propose Semantic Subword Tokenization (SST), which represents historical items as variable-length semantic subwords while preserving fixed-length target decoding. SST first applies Item-level Subword Tokenization (IST) to merge stable adjacent atom tokens into compact semantic subword tokens, thereby reducing intra-item reassembly in the encoder. It then introduces Behavior-induced Co-occurrence Augmentation (BCA) to inject coarse-grained semantic prefix transition signals, guiding the freed modeling capacity toward inter-item behavioral regularities. Extensive experiments on three public datasets and three generative recommender backbones show empirical improvements of SST over fixed-length and transferable variable-length SID baselines. Code is available at https://github.com/mxrcandy/Semantic-Subword-Tokenization.
