Search papers, labs, and topics across Lattice.
This paper reveals a critical evaluation bias in group recommendation methods, where improvements reported on standard benchmarks may not reflect true advancements due to the effects of deterministic tie-breaking in metrics like HR@K and NDCG@K. By applying a tie-aware evaluation protocol that accounts for uniform random tie-breaking, the authors demonstrate that many previously celebrated gains diminish significantly, altering the relative performance rankings of various methods. Additionally, they introduce a temperature-scaled BPR approach that mitigates tie inflation while preserving optimization benefits, underscoring the need for rigorous evaluation standards in this field.
Many touted advancements in group recommendation collapse under tie-aware evaluation, reshaping our understanding of model performance.
Recent group recommendation methods have reported strong improvements on standard benchmarks, but it remains unclear whether these gains always reflect genuine advances in modeling group preferences. In this paper, we show that several recent methods are affected by a systematic evaluation bias caused by the interaction between training-time score compression and evaluation-time deterministic tie-breaking. Specifically, an additional sigmoid transformation before the BPR objective can greatly increase tied top scores, making top-K metrics such as HR@K and NDCG@K highly sensitive to how ties are resolved. We revisit recent representative methods and their baselines on CAMRa2011 and Mafengwo under both group and user recommendation settings, and evaluate them with a tie-aware protocol that computes the exact expectation of HR@K and NDCG@K under uniform random tie-breaking. Our results show that many previously reported improvements shrink substantially under tie-aware evaluation, and the relative ranking of methods can change markedly. We further show that the additional sigmoid may act as implicit margin smoothing during optimization, and that temperature-scaled BPR can retain much of this benefit without inducing severe tie inflation. Overall, our findings highlight the importance of tie-aware evaluation for establishing reliable progress in group recommendation. The code is available at https://github.com/songduoma/TieAwareGroupRec.
