Search papers, labs, and topics across Lattice.
This paper introduces UXBench, a pioneering user-centric benchmark designed to evaluate AI assistants based on real user feedback, addressing the critical need for assessing user experience (UX) beyond mere model performance. The benchmark comprises three tasks鈥擴X Judge, UX Eval, and UX Recovery鈥攗tilizing a dataset of 7,400 instances derived from over 70,000 interaction logs, reflecting diverse user scenarios and failure patterns. Results from extensive experiments on 26 advanced language models reveal that user feedback prediction can be effectively learned, highlighting significant performance gaps and biases in current evaluation protocols, ultimately advocating for a shift towards user-focused optimization in AI development.
User feedback prediction can be learned, revealing critical performance gaps in AI assistants that traditional evaluation methods overlook.
As AI assistants serve millions of users daily, evaluating user experience (UX) beyond general model capability has become increasingly important. We present UXBench, the first user-centric benchmark grounded in real user feedback signals for evaluating preference alignment and dialogue generation. The benchmark consists of three interconnected tasks, UX Judge, UX Eval, and UX Recovery, with 7,400 test instances extracted from over 70K interaction logs of a mainstream Chinese AI assistant. The dataset closely reflects real user distributions, covering 8 scenarios, 83 domains, and diverse failure patterns that pose severe challenges. Extensive experiments on 26 frontier language models provide novel insights into how well models perceive user experience and how improvements in model capability contribute to better dialogue engagement. Through comprehensive analysis of model behavior and performance gaps, we show that user feedback prediction is a learnable capability, where a reward model trained from in-the-wild feedback signals can achieve well-calibrated accuracy. We further document the systematic biases of LLM-as-a-judge evaluation protocols and compare typical response strategies that directly affect user experience. UXBench establishes a new evaluation landscape and calls for greater attention to tailored UX optimization, contributing to a user-centric scaling law that shapes the success of AI assistants.
