Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces τ-Emotion-Bench, a novel benchmark designed to evaluate the robustness of LLM agents in long-horizon tool-calling tasks by incorporating controlled, emotion-conditioned user interactions, addressing the limitation of existing benchmarks that assume cooperative users. The authors also present Tracer, a scalable agentic data-generation pipeline that constructs grounded tasks and collects interaction-aware trajectories, filtering for efficient tool use and user engagement. Experiments reveal that current frontier agents exhibit significant unreliability under emotional user variations, but Tracer-generated supervision effectively improves agent performance across scales and generalizes to unseen domains.
LLM agents falter significantly when faced with real-world user emotions, revealing a critical fragility in their long-horizon tool-calling capabilities.
In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users. Yet existing benchmarks largely assume cooperative and behaviorally stable users, leaving the effect of emotional variation on agent reliability underexplored. We introduce τ-Emotion-Bench, a ten-domain benchmark for evaluating tool-calling agents under controlled emotion-conditioned user interactions while preserving the underlying task objective. To scale both benchmark construction and agent training, we introduce Tracer, a scalable agentic data-generation pipeline that constructs grounded, policy-valid tasks and collects trajectories through an interactive agent--user--environment harness. By allowing dissatisfied users to terminate inefficient interactions, the harness provides an interaction-aware signal that filters trajectories with excessive user dependence and favors effective tool use. Experiments with two user simulators show that frontier agents remain unreliable, with substantial gaps between average task success and consistent task completion. Tracer-generated supervision consistently improves Qwen3 models across scales, generalizes to unseen domains, and transfers to τ-Bench and τ-Trait. Behavioral analyses further show that higher task success is associated with a better balance between tool execution and necessary user follow-up. Overall, τ-Emotion-Bench and Tracer provide a scalable framework for evaluating agent reliability under diverse user behaviors and improving tool-calling agents through interaction-aware supervision.