Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of using large language models (LLMs) as judges for assessing responses generated by conversational recommendation systems (CRS) in the context of music recommendations. By conducting a user study with 20 multi-turn sessions and collecting 400 ratings from domain-expert annotators, the authors find that LLMs align moderately well with human evaluations, particularly in terms of personalization and explanation quality. Notably, LLMs outperform traditional reference-based evaluation methods, highlighting their potential as scalable evaluators in CRS applications.
LLMs can serve as reliable judges for conversational recommendation systems, aligning moderately with human evaluations while outperforming traditional metrics.
Conversational Recommendation Systems (CRS) aim to achieve two primary objectives: recommending relevant items and generating natural language responses. While recommendation accuracy is effectively measured by established ranking metrics, the evaluation of response generation poses a more fundamental challenge. Although human evaluation remains the gold standard, its cost and scalability constraints have motivated the adoption of LLM-as-a-judge as a promising proxy, whose alignment with human judgment in the context of CRS remains an open question. In this paper, we present the first user study to empirically assess the reliability of LLM-as-a-judge for evaluating CRS responses. We sample 20 multi-turn music recommendation sessions and generate candidate system responses using four instruction-tuned LLMs, inducing variance in response quality across model scales. We collect $n{=}400$ ratings from 20 domain-expert annotators, who evaluate each response across two dimensions: Personalization Quality and Explanation Quality. Through bootstrapped correlation analysis, we find that LLM-based judges exhibit moderate positive alignment with human assessments and outperform all reference-based baselines. Furthermore, we analyze how judge performance varies according to model scale and conditioning information, providing practical guidance for deploying LLM-as-a-judge.
