Search papers, labs, and topics across Lattice.
This paper introduces Turing-RL, a novel reinforcement learning framework that utilizes a discriminative Turing reward to train user simulators by evaluating the indistinguishability of generated responses from real user inputs. By focusing on optimizing for indistinguishable responses rather than merely matching ground truth, Turing-RL significantly enhances the performance of user simulators across conversational chat and Reddit forum discussions. The results demonstrate that this approach consistently outperforms traditional methods in both LLM and human evaluation metrics, highlighting its effectiveness in simulating human-like interactions.
Turing-RL reveals that training user simulators for indistinguishability can dramatically improve their performance in simulating human interactions.
Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose {Turing-RL}: a Turing-Test-based reinforcement learning approach for training user simulator models. {Turing-RL} uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains--conversational chat and Reddit forum discussion--we find that {Turing-RL} consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.
MIT CSAIL
Stanford HAI