Search papers, labs, and topics across Lattice.
This study investigates the influence of training seeds on recommender system evaluations by systematically varying the seed while keeping the data partition constant across hyperparameter configurations. The findings reveal that seed variation significantly affects user-level metrics, model selection, and recommendation-list agreement, challenging the assumption that a single seed suffices for robust evaluation. Ultimately, the research underscores the necessity of incorporating training seeds into the evaluation protocol to avoid overstating the stability of results.
Training seeds can dramatically skew recommender system evaluations, revealing that relying on a single seed may misrepresent model performance.
Recommender-system experiments often rely on a single random training seed, assuming that run-to-run stochasticity has limited impact on evaluation conclusions. This assumption is risky, as a training seed may influence several algorithm-dependent mechanisms, including parameter initialization, mini-batch ordering, dropout, masking, latent sampling, and training-time negative sampling. We examine this assumption by fixing the data partition and varying the training seed across hyperparameter configurations. We analyze seed effects at three levels: user-level metric sensitivity, validation-based model selection and recommendation-list agreement. Results show that seed variation is often detectable. Its impact depends on whether configurations are clearly separated, whether validation results transfer to test, and whether similar scores lead to similar top-$k$ lists. Findings suggest that reporting single-seed results can overstate the stability of recommender system evaluation, and that training seeds should be treated as part of the evaluation protocol rather than as incidental implementation noise.
