Search papers, labs, and topics across Lattice.
This paper introduces AgentGym2, a novel evaluation framework designed to benchmark large language model agents in realistic, de-idealized environments that reflect actual deployment challenges. By focusing on agents' abilities to execute end-to-end procedures, discover tools through exploration, and handle noisy inputs, the framework reveals significant performance gaps in state-of-the-art models like Gemini and GPT-5. The findings highlight the inadequacy of existing benchmarks, which fail to capture the complexities of real-world applications, emphasizing the need for more rigorous evaluation methods in AI research.
Even state-of-the-art language models struggle significantly in real-world tasks, exposing critical shortcomings in their deployment readiness.
Language agents, i.e., LLM agents, progress rapidly and are increasingly deployed in production environments. This trend underscores the urgent need for rigorous and realistic evaluations. However, most existing benchmarks evaluate agents in simplified, idealized settings. They typically rely on pre-packaged tool interfaces, overlook critical steps, and assume inputs are clean and fully specified. Consequently, they understate the difficulty of real deployments, where uncertainty and noise are ubiquitous and agents must proactively explore the environment to uncover new tools. To bridge this gap, we present AgentGym2, a new evaluation framework with task instances grounded in real-world end-to-end working demands. Beyond reasoning and planning, it measures agents' ability to execute end-to-end procedures, discover tools via exploration, compose tools for unseen tasks, and remain robust to noisy and underspecified information. Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2, revealing a substantial gap between the capability of current agents and the demands of real-world applications.
