Search papers, labs, and topics across Lattice.
This paper introduces CRAB-Bench and RUSE, two tools designed to evaluate LLM agents in complex, realistic service scenarios characterized by intricate task dependencies and human-like user behavior. The evaluation reveals that even the best-performing LLM agent achieves only 61% pass@1 on CRAB-Bench, with a significant drop in performance when evaluated with RUSE, highlighting the challenges posed by realistic user interactions. Notably, the study identifies that agents struggle with admitting mistakes, often opting for implicit corrections, which underscores the limitations of current LLM capabilities in handling nuanced user behavior.
Even the top-performing LLM struggles with realistic user interactions, achieving only 61% success in complex task scenarios.
Evaluating LLM agents in realistic service scenarios requires complex task dependencies, imperfect user behavior, and an evaluation that accommodates multiple valid solutions. We introduce CRAB-Bench (Constraint-based Realistic Agent Benchmark) and RUSE (Realistic User Simulation Engine) to address this gap. CRAB-Bench generates tasks via a constraint graph over multiple interdependent entities with structured distractors, requiring agents to reason carefully over thousands of misleading candidates where only a tiny fraction of solutions are valid. RUSE replaces cooperative, template-like simulators with realistic users grounded in human behavioral studies, instantiated across diverse personas and four behavioral dimensions. Experiments on four frontier LLM agents show that the best model achieves only 61% pass@1 on CRAB-Bench, and switching to RUSE causes further drops of up to 57%, concentrated in task-solving ability rather than conversational quality. Information Disclosure is the most damaging behavioral dimension, and agents interacting with RUSE are less likely to admit mistakes, instead masking errors through implicit corrections.
CMU ML