Search papers, labs, and topics across Lattice.
This paper introduces AdaptSim, an innovative user simulator designed to enhance the evaluation of Conversational Recommender Systems (CRSs) by addressing key limitations in domain adaptability, user modeling, and evaluation validity. By employing automatic prompt generation and a flexible action mechanism, AdaptSim allows for realistic behavior modeling and diverse linguistic style generation, significantly improving the simulator's effectiveness across various domains. Experimental results show that AdaptSim not only generates realistic dialogues but also provides a robust framework for assessing CRS capabilities through a novel pairwise comparison evaluation method.
AdaptSim transforms CRS evaluation by generating realistic dialogues that adapt to diverse user styles and preferences, overcoming traditional simulator limitations.
Conversational Recommender Systems (CRSs) enhance user experience through multi-turn interactions, yet evaluating their performance remains challenging. While Large Language Model (LLM) based user simulators are effective, they suffer from three key limitations: (1) Lack of Domain Adaptability: Reliance on fixed prompts and predefined action spaces hinders transfer to novel domains; (2) Limited User Modeling: Inability to accurately replicate subtle linguistic styles and dynamic preferences; (3) Insufficient Evaluation Validity: Existing simulators fail to adequately assess fundamental capabilities and system robustness. To overcome these, we propose AdaptSim, an Adaptive domain and automatic prompt tuning User Simulator. AdaptSim offers an efficient framework for evaluating CRSs by enabling realistic behavior modeling and diverse style generation. It leverages automatic prompt generation and an open action mechanism to reduce manual effort and improve cross-domain flexibility. For response generation, we employ controlled text generation with a "think-then-respond" strategy for fine-grained control over language style. For CRS evaluation, AdaptSim incorporates a novel Breadth-First Search (BFS)-based, turn-level pairwise comparison framework for comprehensive assessment. Extensive experiments across three domains and four LLMs demonstrate that AdaptSim generates realistic dialogues, enabling a highly effective and reliable evaluation of CRS capabilities and robustness.
