Search papers, labs, and topics across Lattice.
This study evaluates various recommendation strategies within GESIS Search, a specialized academic search engine for social sciences, by employing the STELLA evaluation framework for continuous, real-time assessment. The research compares traditional lexical similarity, transformer-based semantic document similarity, and session-based recommendations, revealing that users favor semantic similarity approaches over others. Notably, the effectiveness of these recommendation strategies varies by category, indicating that user information-seeking behavior is influenced by the type of information being searched.
Users overwhelmingly prefer semantic similarity recommendations in academic search, highlighting the importance of aligning strategies with diverse information types.
Delivering relevant recommendations in academic search engines is a complex task due to the diversity of subject areas, information types, and user preferences. In this case study, we address these challenges by integrating and evaluating a range of recommendation systems within GESIS Search - a domain-specific search engine for the social sciences that provides researchers with access to research data, publications, variables, and measurement instruments. To support continuous, real-time evaluation of multiple recommendation strategies with actual platform users, we utilize the STELLA evaluation framework. We implement and compare a diverse set of algorithms, including traditional lexical similarity, semantic document similarity by using transformer-based embeddings, and session-based recommendations based on click paths from historical user sessions. Our results show that users prefer recommendations based on semantic similarity, which outperformed term-similarity and session-based methods. However, the performance of recommenders varies across categories within GESIS Search, suggesting that information-seeking behavior differs by information type. Overall, our study provides insights into how continuous evaluation can be incorporated to develop recommendations that better align with the preferences in academic search portals.
