Search papers, labs, and topics across Lattice.
This study critically evaluates the necessity of higher-order sequence modeling in sequential recommendation benchmarks by employing two simple recency-weighted pairwise probes, SeqRules and PCTM. The findings reveal that these probes outperform the widely used eSASRec model on several datasets, indicating that current benchmarks may not effectively capture the benefits of complex modeling architectures. Notably, while the probes excel on some datasets, they also demonstrate limitations, suggesting a nuanced understanding of benchmark efficacy is required in sequential recommendation systems.
Current benchmarks may mislead researchers about the necessity of complex modeling, as simple recency-weighted methods outperform advanced architectures in several cases.
Sequential recommenders increasingly use language-model architectures designed to capture complex, context-dependent interactions. Yet it remains unclear whether widely used benchmarks actually require this modelling capacity. We investigate this question using two simple, recency-weighted pairwise probes that do not learn higher-order sequence representations: Sequential Rules (SeqRules) and our Probabilistic Collaborative Transition Model (PCTM). Using the evaluation protocol of eSASRec, at least one probe exceeds our eSASRec reproduction by 15-38% on three Amazon datasets and by 4.4% on MovieLens-1M, but trails it by 27.3% on MovieLens-20M. On the four remaining datasets, at least one probe also outperforms our sampled-softmax SASRec reproduction by 9-28%, suggesting that these widely used benchmarks are poorly suited to measuring gains from higher-order sequence modelling. More broadly, comparing Transformer-based models against strong recency-weighted pairwise probes provides a concrete test of whether a benchmark can meaningfully measure gains from higher-order sequence modelling.
