Search papers, labs, and topics across Lattice.
This study investigates the user-perceived value and reliability of popularity calibration in music recommendation systems through a user-centric approach. By constructing personalized track lists with varying popularity compositions and analyzing user preferences, the research reveals that while users can distinguish between different popularity distributions, they do not necessarily prefer calibrated lists. Furthermore, the findings indicate that the relationship between JSD-based popularity calibration and user perception is influenced by factors such as item familiarity and the completeness of user history, highlighting the complexities of user experience in recommendation systems.
Users can tell the difference in popularity composition of music recommendations, but they don鈥檛 necessarily prefer the calibrated options.
Popularity calibration in recommender systems has been studied both as a form of user-centered personalization and as an indicator of popularity bias. Most existing work evaluates calibration through offline metrics, often assuming that users prefer recommendation lists whose popularity distribution matches their historical consumption profile. However, user studies on calibration remain limited, and existing findings suggest that calibrated recommendations do not necessarily have a strong effect on user experience. Moreover, although prior work has shown that calibration metrics can correlate with users' perceptions of recommendation lists, the robustness of this relation remains unclear under different levels of item familiarity and incomplete user-history information. In this work, we study the perceived value and measurement reliability of popularity calibration in music recommendation. We construct personalized track lists from users' recent listening histories and use a controlled naive recommender to create lists with different popularity compositions: highpop-heavy, lowpop-heavy, and calibrated. We investigate whether users perceive differences between these lists, whether calibrated lists are preferred, how robust JSD-based popularity calibration is under different familiarity and history-availability conditions, and how computational popularity labels align with users' own popularity judgments. Our results show that users perceive differences in popularity composition, but do not clearly prefer calibrated lists. We further find that the relation between JSD and perceived popularity depends on item familiarity, list composition, and available user history, while computational and user-judged popularity labels only weakly align. These findings contribute to a more critical understanding of popularity calibration as both an offline metric and a user-facing construct.
