Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the limitations of Next-Token Prediction (NTP) by introducing NONTP, which enhances training signal coverage through two auxiliary objectives: Temporal Contrastive Learning (TCL) and Trans-Domain Learning (TDL). TCL aligns hidden states with future trajectories, while TDL enables cross-domain gradient pathways without adding parameters, effectively improving the model's ability to learn longer-range dependencies and utilize multi-domain context. The results show that NONTP significantly outperforms NTP and other benchmarks in both offline metrics and online A/B tests, demonstrating its practical effectiveness in generative recommendation systems.
NONTP boosts generative recommendation performance by over 34% in HR@10 by addressing critical training signal limitations of traditional NTP methods.
Next-Token Prediction (NTP) carries two structural training signal limitations. First, NTP optimizes for single-step prediction only, placing no supervised pressure on learning longer-range behavioral structure -- we term this \textbf{temporal locality}. Second, in multi-domain sequences, each target item embedding receives gradient updates exclusively from the immediately preceding hidden state, with no explicit gradient pathway from cross-domain context -- we term this \textbf{spatial locality}. We propose \textbf{NONTP}, extending NTP's signal coverage along both dimensions through two auxiliary objectives. \textbf{TCL (Temporal Contrastive Learning)} uses a BYOL-style EMA teacher with InfoNCE to align hidden states against a $K$-step future trajectory in representation space. \textbf{TDL (Trans-Domain Learning)} mean-pools cross-domain hidden states and predicts through the shared prediction head, opening a second gradient pathway with no additional parameters. Both are discarded at inference: zero overhead. On a four-domain Meituan industrial dataset (full ranking), NONTP achieves HR@10 +34.3\% over NTP and +18.3\% over MBGR. On the public Amazon Movie-Book-CDs benchmark, HR@10 +2.8\% and NDCG@10 +3.7\%. Online A/B tests confirm CTR +1.8\% and GMV +2.1\% (both $p<0.01$). Ablation studies confirm each component contributes independently, with gradient conflict analyzed as a direction for future work.