Search papers, labs, and topics across Lattice.
This paper tackles the challenge of personalized user satisfaction evaluation in conversational AI by building a conversation satisfaction evaluator that leverages compact user memories and target-turn context. The evaluator is trained to predict satisfaction scores and generate dissatisfaction-oriented rationales. Meta-evaluation demonstrates improved ordinal agreement and dissatisfied-turn detection compared to baselines, and the authors introduce PersTurnBench, a benchmark for evaluating generation models via replay using the verified evaluator.
Stop retraining your satisfaction evaluator for every new model: PersTurnBench lets you compare personalized conversation satisfaction without collecting new user feedback.
User satisfaction with AI assistants is highly personalized: the same response may satisfy one user but disappoint another depending on what each user expects and what they have asked for before. Existing automatic evaluation methods mostly measure generic response quality, making it difficult to judge whether a response satisfies a user at a specific turn. We study this problem as personalized turn-level user conversation satisfaction evaluation. We build a conversation satisfaction evaluator that combines compact user memories with target-turn context to produce satisfaction scores and dissatisfaction-oriented rationales. Meta-evaluation against human satisfaction annotations shows that personalized memory and post-hoc score calibration improve ordinal agreement and dissatisfied-turn detection over supervised, retrieval-based, and generic LLM-as-a-judge baselines. We further introduce PersTurnBench, a personalized turn-level user conversation satisfaction benchmark that uses the verified evaluator to assess generation models via replay. By holding the replay state fixed, PersTurnBench enables controlled comparison of generic generation models and memory-augmented personalized systems without new human labels for every candidate model. The evaluator and benchmark let researchers compare candidate generation models on personalized satisfaction without collecting new user feedback for every model.
