Search papers, labs, and topics across Lattice.
This paper introduces NONTP, an extension of Next-Token Prediction (NTP) that addresses its limitations in temporal and spatial locality by incorporating two auxiliary objectives: Temporal Contrastive Learning (TCL) and Trans-Domain Learning (TDL). By aligning hidden states with future trajectories and pooling cross-domain hidden states, NONTP significantly enhances training signal coverage without adding inference overhead. The method achieves substantial improvements in recommendation performance, with a 34.3% increase in hit rate on a Meituan dataset and notable gains in online A/B tests, indicating its practical effectiveness.
NONTP boosts recommendation accuracy by over 34% by extending training signal coverage without increasing inference costs.
Next-Token Prediction (NTP) carries two structural training signal limitations. First, NTP optimizes for single-step prediction only, placing no supervised pressure on learning longer-range behavioral structure -- we term this \textbf{temporal locality}. Second, in multi-domain sequences, each target item embedding receives gradient updates exclusively from the immediately preceding hidden state, with no explicit gradient pathway from cross-domain context -- we term this \textbf{spatial locality}. We propose \textbf{NONTP}, extending NTP's signal coverage along both dimensions through two auxiliary objectives. \textbf{TCL (Temporal Contrastive Learning)} uses a BYOL-style EMA teacher with InfoNCE to align hidden states against a $K$-step future trajectory in representation space. \textbf{TDL (Trans-Domain Learning)} mean-pools cross-domain hidden states and predicts through the shared prediction head, opening a second gradient pathway with no additional parameters. Both are discarded at inference: zero overhead. On a four-domain Meituan industrial dataset (full ranking), NONTP achieves HR@10 +34.3\% over NTP and +18.3\% over MBGR. On the public Amazon Movie-Book-CDs benchmark, HR@10 +2.8\% and NDCG@10 +3.7\%. Online A/B tests confirm CTR +1.8\% and GMV +2.1\% (both $p < 0.01$). Ablation studies confirm each component contributes independently, with gradient conflict analyzed as a direction for future work.
