Search papers, labs, and topics across Lattice.
This paper introduces DuMateBench, a novel benchmark designed to evaluate autonomous agents in complex, real-world workflows by reconstructing tasks from anonymized user sessions on a production agent platform. The benchmark includes 200 tasks across diverse scenarios, emphasizing the necessity for multi-capability coordination while incorporating real-world complexities such as insufficient resources, instability, and noise. Experimental results highlight significant performance gaps in task completion across various autonomous-agent frameworks and state-of-the-art LLMs, underscoring the influence of environmental factors on agent performance.
Real-world complexities expose significant performance gaps in autonomous agents, revealing that even advanced LLMs struggle with task completion in dynamic environments.
Autonomous agents are increasingly adopted to complete complex, multi-tool workflows in real-world settings. However, existing benchmarks typically separate tasks by application or capability and evaluate agents in environments that are cleaner and more stable than those encountered in practice. We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions collected from a large-scale production agent platform. Each task preserves the relevant pre-solution interaction history, persistent configurations, and workspace state, and is then validated through human verification. The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories, with most tasks requiring multiple capability coordination. We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy, and assess performance using a hybrid deterministic and LLM-as-Judge evaluation protocol. Experiments across five representative autonomous-agent frameworks paired with four state-of-the-art LLMs reveal substantial gaps in strict task completion. Complementary robustness, efficiency, and diagnostic analyses further show that performance under environmental perturbations is jointly shaped by the capabilities of the LLM and the surrounding agent framework. The code and data are publicly available at https://dumatebench.com/.
